Google once faced a fatal problem that prevented it from creating a search index – GIGAZINE

When you want to know something or access a page, you can simply enter the search phrase into your browser and it will instantly return the latest results.Search Engine IndexA complex mechanism incorporating various academic concepts is active, but Google once faced a fatal problem in which the system that creates the index of the search engine stopped, and The New Yorker explains the history and cause of this.

The Friendship That Made Google Huge | The New Yorker
https://www.newyorker.com/magazine/2018/12/10/the-friendship-that-made-google-huge

Google’s core system, which crawls the web and creates a search engine index, stopped in October 1999. Users could still search on Google, but if they searched for something in March 2000, the results were stuck at five months old, and they had problems accessing the latest news and web pages from the search results. In addition to users, there were other important issues. At the time, Google had a contract to provide a search engine index to run Yahoo!’s search engine, and if the system stopped completely, the contract would collapse and Google would even be in danger of running out of funds and disappearing.

Around March 2000, a software engineer at the time of the founding of GoogleCraig SilversteinHe and his team of systems engineers organized an intensive troubleshooting meeting, but after spending four full days going through the code, they couldn’t come up with any improvements. “None of the analysis we were doing made sense,” Silverstein later recalled. “Everything was broken, and we didn’t know why.”

It was once one of America’s leading computer companies.Digital Equipment Corporation (DEC)I joined Google in early 1999.Jeff DeanHe and I joined Google in December 1999 from DEC.Sanjay GhemawatBoth of them dealt with system troubles during their short careers at Google, with Dean later becoming head of Google’s AI division.

Dean and Ghemawat had been good friends since their days at DEC, and apparently liked to work side-by-side writing code together. The two scrutinized the stalled index, spending days finding flaws in the code, such as some words missing or in the wrong order, but were unable to find any obvious bugs. Five days into the project, the two began to suspect that the problem was not software, but physical. They then tried to solve the problem by creating a binary representation of the messy index file.Binary CodeWe tried to understand what the machine sees by simplifying it to:

In binary code, a monitor displays a string of 1s and 0s. Each string represents data in the index, but at one point, Ghemawat noticed that a number that should have been 0 was a 1. When Dean and Ghemawat put together all the words that were misprinted, a pattern emerged. As a result, it turned out that the machine’s memory chips had somehow become corrupted.

Google was founded in 1998, and from 1999 to 2000, when the trouble occurred, it was still being run like a startup, The New Yorker points out. In fact, there were failures and malfunctions in various aspects, such as motherboards, hard drives, and wiring. NASA and financial institutions used special hardware that would not affect the whole even if a single bit of “0” and “1” was flipped due to a hardware malfunction, but Google used cheap computers and could not avoid hardware failures. As a result, Dean and Ghemawat wrote code to compensate for the problematic machines, and a new index was completed, and the countermeasures meeting was dissolved.

The New Yorker points out that the root cause of this problem lies in the origins of Google.Larry PageMr.Sergey BrinHe was not a software engineer, but a search technology researcher enrolled in a doctoral program at Stanford University. According to The New Yorker, Page and Brin’s early software, “BigFiles,” was called “BugFiles” by employees, and when creating important indexing code, they had to start over from scratch if a problem occurred. The New Yorker said, “In Silicon Valley terms, Google was not scalable (a system or business model that can maintain its performance and efficiency as it expands and adapt as needed).”

At the time, Google had a stack of 1,500 devices, each combining a motherboard and a hard drive, but only about 1,200 of them worked because of hardware failures. Because failures caused by hardware failures kept wreaking havoc on the system, Google needed to integrate the computers into a resilient whole. To prevent this from happening again, Dean and Ghemawat wrote code so that the failure of one hard drive would not take down the whole system.Alan Eustace“Paradoxically, to solve large-scale problems, you need to know the details. Dean and Ghemawat understood computers at the bit level,” he said, praising the duo’s abilities.

Google engineers are ranked in stages, with level 1 being IT support staff, level 2 being fresh out of college, and level 3 being master’s degree holders. Level 4 requires several years of contribution or a PhD, and most staff stop at level 5 even after a career. Level 6 engineers are the top 10 percent, level 7 is level 6 with a long track record, level 8 is for those involved in major products or infrastructure, and level 9 is for well-known engineers who are spoken of with respect. Level 10 is a world-leading, influential, and expert executive. And Dean and Ghemawat are senior fellows at Google, the first and only two people at Google to be certified as “level 11”.

Dean and Ghemawat are two people working on one computer.Pair programming“I don’t know why we decided on pair programming. If you can find someone who is compatible with your way of thinking, you two complement each other. I don’t know why more people don’t do it now,” Ghemawat said. It’s rare to find a software engineer with such a good partnership as Dean and Ghemawat, and they are also closely connected not only at work but also in their private lives, including their families. The New Yorker described the incident in which the two of them solved the problem as “the friendship that made Google big.”