Battery Shipping Delay Updates
Last Updated February 12, 2022
Most Recent Updates
02-12-2022 - Items are continuing to be received by delay affected Triad Customers at this time. FedEx is collecting items as they are being released by Customs and processing items out for final destination delivery. Batteries are currently in transit at this time to recipients whose items have cleared safety inspection. FedEx is notifying recipients directly of the ETA with exception of possible weather delays and worker shortages due to omicron variant related illness. Items are currently located within the United States Dept of Homeland Security Central Processing Station at LAX [DHS CPS LAX] airport pending mandatory government safety clearance for common carrier (FedEx) to collect to provide final destination delivery to all remaining receivers at this time. Of over 70 shipments delayed during this time, 8 remain pending for clearance and should be completed within the next days with in transit notifications and ETAs from FedEx to follow directly by email notification to all awaiting delay affected receivers at this time.
As of 04:39 pm EST 02/10/22 Representatives for US Customs Agents in charge have replied to our shipping agent.
"Individual items contained within the parcel block are continuing to process out of DHS CPS LAX to common carrier. The international parcel block of 800,000 items containing your items clearance process at DHS CPS LAX conclusion has changed to 2022-02-14 5:00 LT (UTC -8) or until further notice. Entirety of parcel block is nearing completion for all items contained. Pickup release notifications for collection of cleared items are being sent to authorized carrier as items clear safety protocols. Scancom servers remain offline at this time for vulnerability mitigation. Please await further notice by manual transmission for any pending DG items."
FedEx Ship Manager still has Log4j vulnerability after update.
What is the cause of the delay?
Expert hackers recently implemented a high level "zero day" attack into much of the entire US cyber infrastructure affecting nearly every government agency including Dept of Homeland Security Customs Border Patrol, FBI, NSA, and thousands of public and private companies. This attack is known as Log4j vulnerability. Once this was identified, computers had to be taken offline and all normal operations using massive data systems stopped as you may have noticed the day Amazon went down a couple weeks ago. This has caused a major delay in logistical operations whereby government agencies who regulate and mandate safety protocols for transporting batteries and other international packages depend on computers each day for millions of events to process out shipments. FedEx also does millions of these events. Massive corners of these data systems are offline at this time during this zero day mitigation event.
Log4j Explained - Video
The delayed shipment containing all affected customer battery shipments is currently in the possession of Dept of Homeland Security Customs and Border Patrol (US DHS CBP) agents at CPS (Central Processing Station). The shipment is now in the queue for final inspection and clearance to release all items to the licensed carrier FedEx who will make delivery to the recipient address on file.
This shipment contains batteries for all Triad customers currently affected by the delay caused by the Log4j server vulnerability and backlog of international shipments arriving during year end 2021.
DHS and FedEx servers still remain offline due to the Log4j vulnerability for international small parcels, air express and DG items which includes this group of Triad battery shipments.
Critical information regarding clearance, pickup notifications, and scan tracking data remains being generated manually by phone as the situation remains fluid while DHS and FedEx both work to bring their data centers back online. Their systems will not be brought back online until the complete Log4j system vulnerability has been patched.
The situation has been further complicated with severe weather, flight cancellations and a clearance agent and shipping carrier worker shortage due to mandatory 5 day sick leave time for vaccinated employees who have contracted break through cases of the Covid-19 omicron variant.
We will continue each day to monitor the progress surrounding these important Triad battery shipments and provide updates to this page for your reference. Please be sure to check junk mail and spam folders regularly for direct contact from FedEx shipping notifications once your item is cleared and in transit to your final destination.
Once the Log4j vulnerabilities for international and DG shipments is patched and updated throughout the FedEx Ship Manager system, notifications to recipients by email will process normally.
U.S. Customs is working beyond it's capacity to mitigate, patch and update their system wide vulnerability. They are also working beyond their capacity at this time to clear all all items for release to FedEx for final destination deliveries.
Information regarding shipments will be presented from US Customs agents to the bonded and licensed carrier FedEx only. FedEx will then notify recipients directly by email only. Outbound phone calls by FedEx to recipients for these tracking updates is temporarily suspended at this time.
We sincerely apologize to all our valued clients at Triad Electric Vehicles for this unforeseen and unprecedented delay. We are doing all we can to retrieve real time information and work with the agents in charge to clear these important shipments for all who have been affected at this time.
This is an unforeseen anomalous logistics complication and in no way is considered normal business operations. Battery shipments were in fact sent in advance of vehicle production to account for mandatory safe transit clearance times. Battery shipments have a normal clearance time of 7-10 days at Dept of Homeland Security Customs Border Control. It’s the law under IATA and IACO UN Regulations that they are the governmental regulatory agency who must clear this powerful class of battery to keep delivery drivers and consumers safe while items are transit according to DOT, FAA and US Labor laws.
FedEx is the licensed carrier who makes delivery of this class of safety cleared battery shipment. Once cleared by DHS then it gets released by them and FedEx is notified to collect for pickup, scan the item for tracking purposes and then the item immediately begins the final destination transit process which has a normal time frame of 3-4 days to the door of the receiver. Battery shipment total normal timeline is 12-14 business days, which is on target with the same arrival time of the primary Triad vehicle shipment.
FedEx Weather and other service disruptions
FedEx Operational Impacts
Winter Storm Landon
February 6, 2022
Winter Storm Landon has caused disruptions that continue to affect FedEx operations across the Central United States. The storm continues to create potentially hazardous operating conditions and the safety of our team members remains our number one priority. Contingency plans are in place, and we are prepared to provide the best possible service as local conditions allow. We will continue to monitor the situation to minimize the impact on service.
Contingency plans are in place, and we will be prepared to provide the best possible service in areas affected by the winter storm and as local conditions allow.
Note that potential service disruptions may not affect FedEx Express, FedEx Ground, FedEx Freight, FedEx Office, etc. the same. This may result in different levels of impact as well as cities, states and ZIP Codes serviced.
Super Bowl Sunday Feb 6 - Feb 13
Hey football fans! Our game face is on as we prepare to deliver the Vince Lombardi trophy for Super Bowl LVI. Meanwhile, we’ll continue to operate in the Los Angeles metro area as thousands of NFL fans head to SoFi Stadium to enjoy the festivities. Service to some customers in the area may be impacted from February 5 to February 13 because of government security measures in place for Super Bowl LVI.
We are implementing contingency plans and adjusting operations to minimize delays while continuing to provide the best possible service. We encourage all customers shipping into the area to check with your recipients to understand their locations relative to the Super Bowl activities. Locations in and around SoFi Stadium as well as in the following ZIP Codes may be impacted:
FedEx Express: Los Angeles, CA metro area – 90014, 90015, 90017, 90045, 90250, 90301, 90302, 90303, 90304, 90305
FedEx Ground: Los Angeles, CA metro area – 90301, 90302, 90303, 90304, 90305
The explosive surge of the COVID-19 Omicron variant has caused a temporary shortage of available crew members and operational staff in the FedEx Express air network. The health and safety of our team members is our top priority. We are implementing contingency plans and adjusting operations to minimize delays while continuing to provide the best possible service to our customers during these difficult times. Economy Domestic FedEx Express Freight (FedEx 2Day Freight and FedEx 3Day Freight) is currently suspended.
PLEASE READ THIS ENTIRE STATEMENT
US CYBERSECURITY & INFRASTRUCTURE SECURITY AGENCY (CISA) ISSUES EMERGENCY DIRECTIVE
A malicious, self-propagating program known as “Log4j” has affected millions of data servers across the USA encompassing all US Federal agencies including US Dept of Homeland Security Customs and Border Patrol, FBI, CIA, NSA and the US Postal Service, and huge data companies such as Amazon, Google, Oracle, Microsoft, FedEx, and thousands of other corporations, agencies and organizations.
To be clear, Triad Electric Vehicles data is unaffected is completely secured as we do not store data online or over cloud-based servers specifically out of extreme caution over privacy protections.
However, the Log4j data breach is a global disaster preventing FedEx from presenting tracking information on international shipments as they work to resolve the issue. Both United States Department of Homeland Security Customs and Border Patrol (US DHS CBP) and FedEx must complete their computer system mitigations before they can repair their data systems and resolve the issues to be able to provide electronic communications for tracking details on smaller international parcel sized shipments. Specifically, and generally this refers to Triad battery shipments at this time.
During this time, due to these extraordinary circumstances beyond our control, we unable to collect tracking details for international deliveries of small parcel packages due to Log4j vulnerabilities to USDHS CBP, FedEx, UPS and USPS data and tracking servers.
The items in transit to you remain in transit to you at this time using offline systems right now, however the tracking information and scans normally collected and stored on FedEx and USDHS CBP computer servers is unable to be presented by these companies and government agencies while they mitigate this historically massive vulnerability at a critical time of year for shipping.
Log4j is a program that was embedded into massive Apache servers by hackers and allows bad actors to access and take control of those computer systems remotely for complete and full control of these computer systems. Simply put, this is an unimaginably huge data breach where hackers have exploited a very small vulnerability in software running on millions of computer servers all throughout the entire US data infrastructure system.
It’s been devastating to the U.S. supply chain during this critical time and companies and the U.S. federal government are doing all they can to mitigate the largest zero day attack in modern history.
Tens of thousands of federal and private programmers are scrambling to solve the issue, however once they fix one problem it seems another one is created. These are really big data servers, shared systems owned by Amazon Web Services, Google Cloud, Alibaba Cloud, Oracle Cloud, IBM Cloud and others which are leased by the U.S. Federal Government and other U.S. big companies for data storage used for thousands of data points per second.
The DHL tracking system for Triad electric vehicle deliveries DOES NOT have any indications of being affected. Vehicle delivery tracking remains unaffected. FedEx is the carrier who holds the license for transporting high powered battery shipments after clearance from U.S. DHS CBP agents.
The Fedex and U.S. Customs systems used for battery safe transit clearance has been compromised by Log4j vulnerabilities. https://www.fedex.com/en-us/shipping/ship-manager/software.html#tab-4
Triad is calling DHS CBP representatives and FedEx on your behalf each day for any information they can provide, which may include details about their data security patches, offline updates about shipments arriving, updates to existing tracking numbers for server groups that have been repaired, additional service delays, or new tracking numbers for shipments in transit at this time that are in transit to final destinations. We will provide updates to this page as soon as FedEx and US DHS CBP Federal Agencies involved in safe transit clearance of batteries (and other small international air parcel packages) are able to respond to our daily requests.
As of January 7th, 2022 Triad managers communicating with representatives of DHS CBP have confirmed all items are in possession of the CBP Clearance Agents at LAX at this time; although DHS CBP representatives stopped short of providing an exact date they will complete their releases for pickup notices to FedEx in order to collect all items in their possession at this time.
DHS representatives however did provide an estimated date of release based on their work load at this time as January 14th, 2022 until further notice. If upon January 14th the releases are completed, FedEx will be notified to collect all items and scan them in locally to update the tracking details.
As of January 7th, 2022 FedEx trace managers also have stated the tracking numbers may change at the DHS pickup site upon release and the will make every attempt possible to update the receivers manually by email directly, although it is possible the packages may arrive without prior notice.
FedEx Ship Manager v.3509 Log4j – Service Bulletin Board Message
Due to the urgency of this issue, FedEx recommends upgrading immediately to version 3509 to consume the Apache Log4j 2.16.0 release to address the vulnerabilities for Java 8 users.
FedEx is actively assessing the situation related to the Log4j Remote Code Execution vulnerability.
FedEx Ship Manager™ Software (GSM)
Versions 13.62 and 14.56
Upgrade to version 14.57 upon release in January 2022
The above fixes will remediate the known issues associated with the Apache log4j vulnerability; any subsequent issues will be addressed as needed.
What is the risk?
According to Apache, some Log4j versions of this exploit are ranked 10 out of 10, with a score of 10 representing the most extreme vulnerability. This means an unauthenticated remote actor could exploit this vulnerability to take control of an affected system. To learn more about the risks and ranking click here.
We encourage customers to follow security best practices including those recommended by Apache (Apache Log4j Remote Code Execution), in addition to upgrading FedEx solutions as provided. Details are listed below:
FedEx Ship Manager® (FSM) Versions 340x and above
Upgrade to version 3509 to consume the Apache Log4j 2.16.0 release to address the vulnerability for Java 8 users.
For any additional related questions or the most updated information, customers should contact their Customer Technology representative.
FedEx Delays: Customers frustrated
Official Statement from FedEx "We understand our customers’ frustrations and apologize for any inconvenience as we accelerate contingencies to resolve delays caused by significant package volumes, IT vulnerabilities, and an ongoing industry labor shortage. FedEx is committed to providing service to the best of our ability as local conditions allow. We will continue to monitor these situations in an attempt minimize the impact on service."
Government agencies and (including but not limited to) FedEx, USPS, and many other companies are taking their computer systems offline at this time as a precaution in response to the Log4j vulnerability. CISA says the temporary takedown reflects the extreme risk and urgency of patching the flaw.
Homeland Security secretary Alejandro Mayorkas said on Thursday that he is "extraordinarily concerned" about the vulnerability.
More Facts about Log4Shell (CVE-2021-44228) Vulnerability
- Originally reported on over 1 year ago with no investigation by any U.S. Government agency or mitigation by any U.S. company.
- Privately disclosed to the U.S. Apache Software Foundation by China’s Chen Zhaojun of the Alibaba's Cloud Security Team, again November 24th.
- U.S. Apache Software Foundation publicly disclosed to their U.S. clientele including the Federal Government on December 9th
- The issue was not addressed by any Federal or civilian organizations until December 17th, 2021
THE DHS CYBERSECURITY & INFRASTRUCTURE SECURITY AGENCY (CISA) ISSUES EMERGENCY DIRECTIVE REQUIRING FEDERAL AGENCIES TO MITIGATE APACHE LOG4J VULNERABILITIES
The Department of Homeland Security's Cybersecurity and Infrastructure Security Agency issued an emergency directive on Friday December 17th, 2021 that all federal and federal civilian agencies must assess their systems for mitigations related to the Log4j vulnerability by December 23, 2021.
CISA has determined that this vulnerability poses an unacceptable risk to Federal Civilian Executive Branch agencies and requires emergency action.
This emergency action is based on: (1) the current exploitation of these vulnerabilities by threat actors in external network environments, (2) the likelihood of the vulnerabilities being exploited, (3) the prevalence of the affected software in the federal enterprise, (4) the high potential for a compromise of agency information systems, and (5) the potential impact of a successful compromise.
Official USPS statement: “Computer networks are constantly under attack from criminals who try to exploit vulnerabilities to illegally obtain information. Similar to other companies, the Postal Service’s Information Security program and the Inspection Service uses industry best practices to constantly monitor our network for suspicious activity. Out of an abundance of caution, the Postal Service is further investigating to ensure that anyone who may have sought to access our systems inappropriately is pursued to the fullest extent of the law.”
As a valued customer of Triad Electric Vehicles, you can be assured that we will continue to contact our bonded shipping agents at origin and our FedEx agents here domestically on your behalf each day for manual updates.
Once the release is provided by DHS manually to FedEx as servers are offline, you will immediately be informed by email. FedEx managers have manually entered auto updates to receivers for their shipment(s) in transit at this time.
**NOTE: Because of the Log4J server issue affecting U.S. Customs and FedEx tracking scan databases, it is entirely possible that battery shipments may arrive unannounced without tracking because they are using offline systems at this time. FedEx is changing tracking numbers throughout the delivery process with handheld label printers at pickup and transfer stations without notifying us of tracking number changes. FedEx is currently utilizing smaller node intranet systems while their larger internet systems remain offline as they mitigate the Log4jShell vulnerability.
If items do arrive unannounced without tracking at this time, please let us know by sending a brief email with the final tracking number or a photo of the outer box packaging so we can cross reference this information with our FedEx battery shipping agents who are working hard during this time on your behalf to remain in contact with the U.S. Customs agents who clear all battery shipments for safe transit.
Federal Agents must by law clear all high powered battery shipments for safe transit before carriers can collect items for final delivery. More resources about the compounded delivery delays affecting FedEx and U.S. Customs clearance processes at this time is below.
Additional information affecting possible delays of battery shipments at this time.
Recent Updates about the container ship backlog at U.S ports
Inadequate CBP Staffing
DHS expands paid program to hackers to hunt down Apache vulnerability
Commercial airlines are commonly used by shipping companies to transport goods throughout the U.S supply chain.
Airlines are struggling with cancellations and delays due to severe weather and staffing shortages caused by the omicron variant.
US airlines say the disruption is due to crews testing positive or isolating.
FedEx Corporation FDX is taking a hit from Omicron-induced spike in coronavirus cases and adverse weather conditions across the United States. https://www.yahoo.com/now/fedex-fdx-warns-delays-amid-160004030.html