Artificial Intelligence (AI) is developing at a pace humanity has never experienced before. It is no longer advancing linearly but exponentially, with each generation of AI helping accelerate the development of the next.
We have rapidly moved from generative AI that answers questions and makes videos to agentic AI that reasons, plans and acts, and towards Artificial General Intelligence (AGI) and Super Intelligence (SI) that will equal or surpass human capabilities across virtually every intellectual domain.
The paradox is frightening. AI is fast becoming the greatest instrument of prosperity ever created, yet some of the very people developing it are warning that it could also become one of humanity’s greatest threats. So, damned if we do, damned if we don’t.
The latest warning comes from Bill Gates, traditionally one of technology’s greatest optimists. Gates recently warned that AI is powerful enough to trigger events that could cause as many as a billion deaths if placed in the wrong hands. He argues that industry self-regulation is insufficient and that governments and law-enforcement agencies must become involved in safeguards and monitoring. His concerns include AI-enabled cyberattacks, biological threats and other forms of catastrophic misuse.
Even more disturbing are warnings coming from inside the AI industry itself. Evan Hubinger, who leads alignment research at Anthropic, recently said that he personally believes there is a greater than 10 per cent probability that AI could kill all humans within the next decade. This is not a scientifically established probability, but the personal risk assessment of a researcher whose work focuses on keeping advanced AI aligned with human intentions. Former Anthropic and OpenAI researcher Jacob Coxon has voiced similarly grave concerns.
And the warnings are no longer entirely theoretical. During cybersecurity evaluations, hundreds of OpenAI AI agents reportedly escaped the intended boundaries of their testing environment, discovered ways to communicate with one another and formed what they themselves described as a ‘collective’ or ‘swarm’. They eventually compromised systems belonging to AI platform Hugging Face. OpenAI called the incident a ‘warning shot’, acknowledging that increasingly capable agents can circumvent technical controls, collaborate through unintended channels and take actions humans did not specifically direct. This is particularly significant because AI is becoming increasingly autonomous.
OpenAI’s GPT-6 Astra represents another major leap. OpenAI describes Astra as its most capable broadly deployed model and the first to reach the Critical level for cybersecurity capability under its own Preparedness Framework. With appropriate tools and access, it can identify previously unknown vulnerabilities and develop ways of exploiting sophisticated systems without continuous human guidance. OpenAI consequently introduced stronger isolation, monitoring and security measures around the model.
At a White House meeting on September 29, 2026, President Donald Trump brought together leaders from OpenAI, Anthropic, Google, Meta, Nvidia and other technology companies. They agreed to voluntary safety standards, including internal controls and independent reviews. Trump favours keeping America ahead in the AI race and avoiding regulation that could surrender technological leadership to China. The dilemma is obvious: regulate too aggressively and a country may lose the AI race; regulate too little and increasingly powerful systems may develop faster than society can safely control them. He also rebranded AI as SI.
Yet there is another side to this extraordinary technology. Elon Musk told the same White House gathering that the most likely outcome of AI (SI) is an “age of abundance”. SI could revolutionise medicine, education, science and productivity. Gates himself, despite his warnings, has emphasised AI’s enormous potential in healthcare, agriculture and education, particularly in developing countries. His foundation has announced a $1 billion initiative supporting such applications.
We are therefore crossing an important threshold. The question is no longer simply what AI can tell us. It is increasingly what AI can do without us. Stopping AI globally is increasingly unrealistic. Ignoring its dangers would be equally irresponsible. The rational course is neither blind acceleration nor technological retreat, but accelerated capability accompanied by intelligent safeguards. The danger is not only AI becoming more intelligent; it is human intelligence becoming less exercised.
Advanced countries possess trillion-dollar technology companies, enormous computing infrastructure, frontier laboratories, semiconductor capabilities, venture capital and governments capable of investing hundreds of billions of dollars. They can afford to develop AI and invest heavily in controlling its risks at the same time. Pakistan cannot.
I have argued repeatedly in these pages that AI is no longer simply another technology. It is becoming the infrastructure upon which economic power, national security, education, industry and ultimately global influence will rest. Countries that develop, control and commercialise AI will increasingly shape the twenty-first century; countries that merely consume technologies developed elsewhere risk becoming technologically and economically dependent.
This presents Pakistan with an especially difficult choice. We cannot afford to stand on the sidelines. Universities must teach students how to work with AI without surrendering human intelligence to it. Critical thinking, creativity, ethics, judgement, and independent reasoning become more important, not less, when machines can instantly generate essays, software, research, analysis and theses.
Pakistan must therefore pursue two objectives simultaneously: AI capability and AI responsibility. We need serious national expertise in AI safety, alignment, ethics, privacy and cybersecurity. We cannot simply import AI systems and assume that safeguards developed in Silicon Valley will automatically protect Pakistan’s national interests. The coming contest will not simply be between humans and machines. It will be between countries that understand, build and control intelligent technologies and those that depend upon them.
For Pakistan, standing still may ultimately be the greatest risk of all. We may indeed be damned if we do and damned if we don’t. But if the age of Super Intelligence is approaching, Pakistan cannot afford to enter it unprepared.
