While this data on the population is currently stored in siloed and disparate databases, connecting it could make it possible to automatically follow individuals' records across all of the Home Office's many directorates, from the two years' worth of car journeys logged in the ANPR data centre, to the passports database, the police databases, and many others.
After laying off over a third of its old IT staff, the Home Office has recently been attempting to recruit Hadoop specialists to help it build and maintain this new single platform , with a presentation and talk seemingly doing the rounds around the user circuit until the Home Office got spooked by The Register.
The other speaker at the HUGUK meeting, the head of strategy and architecture, Simon Bond, recognised this and offered a slide suggesting the scale of those databases.
The TPT's crucial work , as a Home Office spokesperson described it, included taking greater direct control over the design, delivery and operation of technology systems; standardising, integrating and reusing solutions across services and developing a broader supplier base, including niche expert suppliers.
Such niche expert suppliers are likely to include San Jose-based Hortonworks.
At that time Hortonworks was the only accredited Hadoop support company listed on the government's procurement platform G-Cloud and so the contract was awarded to them without the contract tender for a proof-of-concept going public.