In networking, like life — timing is everything
Every computer today has a variety of clock styles to choose from, be it analog, digital, or a Big Ben lookalike. For most users, the basic clock’s accuracy is likely sufficient, even if it is off by a minute or two.
However, for enterprise software applications and related processes, highly accurate and synchronized time is a necessity. An inaccurate computer clock can cause significant problems. A minute or two discrepancy could be a significant and unacceptable margin of error since many applications require that the time be kept accurate to the nearest second or less.
For example, computers in financial institutions must keep highly accurate records of when a transaction was completed. Similarly, software used in the manufacturing process requires mixtures to be executed at a precise time.
The Internet, radio, and TV stations also need computers that can switch feeds or link up with remote links at the correct time. When the time on enterprise network devices is incorrect, the effects can be costly and significant.
An accurate time source, as well as time synchronization between two devices, is a necessity. However, clocks on computers cannot be depended on for this source because of their propensity to drift. They use oscillator circuits or a battery-driven, quartz crystal clock (mainly for cost savings), which can drift up to minutes per day. With that, serious timing errors can quickly occur.
In addition to inaccurate clocks, an organization needs to defend its timing infrastructure against malicious attacks from internal and external attackers. One of the ways an attacker will try to hide their tracks is by modifying the clock on systems they have breached.
Steven Teppler, Esq., a well-known lawyer in the technology space, astutely notes that there are regulatory imperatives to maintaining accurate records. However, emerging legal discovery and evidence management court decisions are beginning to impose severe sanctions and penalties on parties in lawsuits who engage in time-based data manipulation.
Teppler says the implication is that time must be accurate and synchronized to the extent possible within the network, and that this accurate and synchronized time must also be reflected in its association with enterprise computer-generated records in a manner also sufficient to withstand a legal challenge.
Perhaps the most significant benefit of effective time synchronization is that it will not make IT look foolish. The picture of the Boeing 757 hitting the Pentagon on the morning of September 11, 2001, is one of the most heartbreaking pictures of the last decade. Regrettably, the time stamp on the video was “September 12, 2001, 5:37 PM.”
The picture, unfortunately, is used extensively amongst the 9/11 conspiracy community. Having the correct time on the video monitors would have obviated such misuse.
This white paper looks at the need for accurate and synchronized enterprise time, what products are available to provide this capability, and how to implement time synchronization.
Need and Risk
Doing things on time is a frequent requirement as many activities need to be synchronized with others to operate at peak levels. However, the reality is that synchronized time is a relatively new phenomenon, as it was just 125 years ago, on November 18, 1883, when Standard Time was created. Before 1883, the local mean time was used throughout the USA, which resulted in a plethora of local times. This alone caused chaos to train schedules, with travelers often missing their trains.
Effective time synchronization can also show if improprieties have occurred. Perhaps the most significant case where time synchronization could have helped–or prevented–Enron exemplifies fraud. The CEO and CFO of Enron made a habit of engaging in time-based data manipulation. CFO Andrew Fastow and his team altered financial data to suit whatever it was they wanted the investing public or government authorities to know, or not know.
Specifically, Fastow backdated documents to manipulate Enron’s financial statements and, as a result, drained millions of dollars that rightfully belonged to Enron and a bank that invested with Enron. He also backdated documents to overstate the value of a technology company in which Enron had invested.
Enron is not an isolated case. Many other companies, including NextCard, Autotote, RiteAid, Parmalot, and Adelphia, acted in similar ways. In all of these cases, effective time synchronization would have provided data integrity assurance of financial reports, grant letters, loan reports, securities transactions, letters of credit, and much more.
The importance of accurate time and time synchronization is two-fold, as it:
· allows events to occur at the proper time via event synchronization. In this way, an organization can schedule a process and ensure that it starts or stops on time, or runs for a specified period regardless of when it starts or stops. This ensures that cooperating processes can interoperate correctly so that if one process hands a task off to a second process, that process will be ready to accept the handoff.
· provides proof when events occurred or did not occur, in other words, using time as a critical feature of digital forensics. If IT does not have synchronized time, it is essential to determine the associated risks. Organizations need to know how accurate their clocks should be–be it minutes, seconds, or milliseconds. Do not underestimate the risks of inaccurate time; if you do not practice due care pertaining to the time on your network and application, the organization can be legally liable for negligence.
Given the legal, practical, and operational realities, adding time services functionality to your enterprise network is no longer an option. The beauty of implementing a time services infrastructure to your organization is that it will not break the bank. The approximate cost varies between $2,000 and $10,000 depending on the level of accuracy required, and if redundancy is needed.
The time server infrastructure itself initially can be up and running in a day but will take longer (exactly how long is dependent on the organization and requirements) to deploy fully. Some of its many benefits are:
- Reduced downtime
- Prevention of operational failure
- Avoidance of data loss
- Improved security
- Mitigation of legal exposure
- Time services ROI often measured in weeks or months
Here is a practical example: An attacker illegally infiltrates your system on Wednesday, July 22, 2020, between 16:38:39 and 17:25:37. Your system logs show that these events occurred starting at 19:49:12. The attacker has a dozen witnesses stating that he was with them watching the 2019 World Series highlights from 18:00 to 21:00. Most prosecutors would not take the case as the logs cannot be admitted as evidence.
Regulatory Imperatives for Time Synchronization
From a regulatory perspective, more and more industry standards are requiring time synchronization. Some of these standards and standards-making bodies are:
- 21 CFR Part 11
- Payment Card Industry Data Security Standard (PCI DSS)
- European Telecommunications Standards Institute (ETSI)
- National Emergency Number Association
- Public Safety Answering Point Master Clock Standard
- National Fire Protection Association
- Standard #1221 — Installation, Maintenance and Use of Emergency Services Communication Systems
One of the most detailed specifications around time synchronization is the PCI DSS version 3.2.1. Section 10.4 states: Using time-synchronization technology, synchronize all critical system clocks and times and ensure that the following is implemented for acquiring, distributing, and storing time.
The PCI testing procedures for requirement 10.4 are to obtain and review the process for acquiring and distributing the correct time within the organization and the time-related system-parameter settings for a sample of system components. You should verify the following are included in the process and implemented:
- A known, stable version of NTP (Network Time Protocol) or similar technology, kept current per PCI DSS Requirements 6.1 and 6.2, is used for time synchronization.
Internal servers are not all receiving time signals from external sources. [Two or three central time servers within the organization receive external time signals directly from a special radio, GPS satellites or other
- external sources based on International Atomic Time and UTC (formerly GMT), peer with each other to keep accurate time, and share the time with other internal servers.]
- Specific external hosts are designated from which the time servers will accept NTP time updates (to prevent a malicious individual from changing the clock). Optionally, those updates can be encrypted with a symmetric key, and access control lists can be created that specify the IP addresses of client machines that will be provided with the NTP service (to prevent unauthorized use of internal time servers).
The implications of PCI non-compliance are significant, from fines levied by Visa and MasterCard to having your payment processor charge higher fees, negative publicity, and more. Non-compliance is risky, costly, and can quickly bankrupt a merchant.
Network Time Protocol (ntp)
No discussion about time synchronization would be complete without the mention of the Network Time Protocol. NTP has been in use for over 40 years and remains the longest-running, continuously operating, Internet application protocol.
NTP is a User Datagram Protocol or UDP-based protocol. With UDP, computer applications send messages, known as datagrams, to other hosts to set up special transmission channels or data paths without requiring prior communications. UDP is an unreliable protocol, and is used for service and speed, but not for the reliability or data integrity.
NTP was designed to synchronize the clock on a client device with the clock on a network time server. Note that NTP is simply the protocol, and the use of NTP requires separate client and server applications.
NTP is roughly accurate within 10–100 milliseconds, and even though it uses UDP, which is an unreliable protocol, it has been architected to sustain accuracy and robustness, even when used over numerous gateways and their respective delays. NTP explicitly determines the offset of the client’s clock relative to the time server’s clock. The client then sends a UPD time request packet to the server, which is time-stamped and returned. The NTP client computes the local clock offset from the time server and makes an adjustment.
The use of NTP can be broken up into the following five steps:
- NTP Design — Choose your NTP time source, either Internal (more control, more management) or External (less control, less management).
- NTP Topology — Issues include the desired level of time accuracy, number of NTP clients, network infrastructure redundancy and network physical topology and geography. Investigate how the sites are connected as round trip delays can impact NTP and negatively affect time accuracy.
- Feature evaluation — Determine which NTP features to use, basic security (authentication, access control) and redundancy (redundancy between peers, redundancy configuration on clients).
- Management — How much you need to manage your NTP infrastructure is dependent on how vital synchronized time is to your organization.
- Audit — Your time infrastructure must be able to prove that the time on any monitored system was correctly synchronized at a particular time and date with a specified time source. Industry-specific regulations often require this. Note that the audit logs must be used within the context of digital forensics. Your staff needs to know and follow the rules of evidence.
Some organizations are reluctant to use NTP given the requirement to punch yet another hole in their firewall to allow NTP port 123 through. The primary concern is that hackers will use port 123 as a point of entry to conduct extensive network attacks. An additional concern about opening port 123 is that it can provide information about the network and serve as an avenue for attack. Some of the information that can be gathered from port 123 includes:
- System uptime
- Time since reset
- Time server packets
- I/O, memory statistics
- NTP peer list
The attacker can also run a replay attack using captured packets or stop security-related chronograph (cron) jobs from running or cause them to run at incorrect times. For that reason, many organizations prefer to use a GPS-based synchronization system.
GPS satellites have atomic clocks and GPS-based time servers synchronize with those clocks, which are accurate to approximately one-millionth of a second to UTC. Since this occurs behind the corporate firewall, there is no need to open it to another protocol.
Time Synchronization Checklist
The following time synchronization checklist is an excellent way to start things rolling:
- Manually ensure that all firewalls, routers, critical servers, etc. have the correct time.
- At this point, synchronizing by calling the United States Naval Observatory Master Clock at 202/762–1402 is sufficient.
· Identify all critical network devices in your organization that requires accurate time.
· Appoint a responsible technical staff member to be the time services liaison and to manage time services.
· Meet with vendors of time synchronization equipment to determine the solution that best fits your organization and specific needs.
· Advise the CIO and CISO on the security risk of non-synchronized time.
· Get management approval for the purchase of time synchronization equipment.
· Work with the CIO and CISO to ensure that time synchronization is an enterprise policy.
Time synchronization must be made part of the corporate IT systems and security policies. For example, the following policy is quite effective: “Time synchronization to an accurate time source is required on all enterprise network devices.” Without a policy, there will be no impetus for staff to achieve the goal of accurate, synchronized time.
Time Synchronization Products
For those companies interested in using a time synchronization appliance, several vendors are offering state-of-the-art capabilities. Three leading vendors are:
All of these vendors’ products have roughly the same functionality, although each has its own strengths. It is crucial, though, to focus on your specific requirements first, rather than focus on the feature set of each appliance.
All major vendors have stratum 1 NTP/NTP time servers that use GPS via oven-stabilized crystal oscillator (OCXO) and rubidium oscillators. These maintain time standard if time reference is lost and have a dial-out modem that provides back up to GPS or functions as the primary reference, such as for disaster recovery, and has accuracy within a few microseconds over a heavy load.
Finally, do not forget that you must secure the time appliance itself. There are many ways in which this can be done.
Some of the most effective security features to protect a time server or appliance are passwords, SSL, access control lists. Use all of these for maximum security and protection of the device.
Finally, realize that while time synchronization hardware is relatively inexpensive for most organizations, it may be a cost factor in some. Determine how much your organization can afford to spend.
The need for synchronized time is a crucial business and technology requirement. As such, it is an integral part of an effective network and security architecture. Ensuring accurate time is relatively inexpensive and offers a significant return on investment. It is also a great way to be in compliance with your various regulatory efforts and to stop your company from getting negative press.
As organizations and IT processes become even more highly synchronized, the importance of network time synchronization will only increase, and so will the need for accurate, synchronized time.