I’ve been thinking about the future of software and machine learning and reached some conclusions. Conclusions others have reached as well.
Machine Learning has started to converge on a few models and as time goes on this will narrow further. These models will live on specialized hardware and we will compile different, less resource intensive ones to ship on standard machines.
What this means is people will spend far more time on data. You will program the prompts you feed your GPT-N for text generation, you will carefully curate which samples you use to distill and prompt your GAN/VAE, you will bootstrap your own dataset using pre-existing models, you will write programs to generate a dataset, and on and on.
A lot of software is, or should be, very modular with several tiers of abstraction. For example LLVM takes your code and creates an intermediate graph which then gets optimized and compiled. As ML improves it will eat these layers of abstraction. Code will be written for humans to read but graph-based RL agents – or whatever else comes along – will optimize your program to run optimally on any arbitrary hardware.
So what should someone, who wants to make a career in software – while humans still have a hand in its creation – focus on now and in the future?
-
The very best models are and will continue to be creatures of scale. Someone will have to manage the dataflow. So, some level of knowledge on massive networks, big data pipelines, and parallel computing.
-
Actually gluing together the different models and techniques that compose your program. How to compose the models to get your desired results. This will probably look like traditional software engineering, with calls to certain functions semi-black box calls to ML models.
-
Learning how to program your data and bootstrap a dataset. Examples of current techniques are prompt programming with GPT-3, using Yolo to noisily annotate a bigger dataset, unity Agents. I’ve seen a lot of odd examples here too, including a math dataset for proofs.
-
Very deep domain knowledge in a few subjects so that you can adapt the more universal model to your niche case.