Battery Shipping Updates
Last Updated January 22, 2022
Most Recent Updates
As of 03:25 pm EST Representatives for US Customs Agents in charge have replied to our shipping agent. "Individual items contained within the parcel block are continuing to process out of DHS CPS LAX to common carrier. The international parcel block of 800,000 items containing your items remains expected to conclude clearance process at DHS CPS LAX by or before 2022-01-28 08:00 LT (UTC -8) or until further notice. Pickup release notifications for collection of cleared items are being sent to authorized carrier as items clear safety protocols. Scancom servers remain offline at this time for vulnerability mitigation. Please await further notice by manual transmission for any pending DG items."
As of 10:17 am EST Representatives for US Customs Agents in charge have replied to our shipping agent. "The international parcel block of 800,000 items containing your items remains expected to conclude clearance process at DHS CPS LAX by or before 2022-01-28 08:00 LT (UTC -8) or until further notice. Some of the individual items contained within this parcel block have already processed out of DHS CPS LAX to common carrier. Pickup release notifications for collection of cleared items are being sent to authorized carrier as items clear safety protocols. Scancom servers remain offline at this time for vulnerability mitigation. Please await further notice by manual transmission for any pending DG items."
As of 09:12 am EST Representatives for US Customs Agents in charge have replied to our shipping agent. "Individual items contained within the parcel block are beginning to process out of DHS CPS LAX. Pickup release notifications being sent to authorized carrier on clearance. Scancom servers remain offline at this time for vulnerability mitigation. Please await further notice by manual transmission for any pending DG items."
As of 11:20 am EST Representatives for US Customs Agents in charge have replied to our shipping agent. "The international parcel block of 800,000 items containing your items is expected to conclude clearance process at DHS CPS LAX by or before 2022-01-28 08:00 LT (UTC -8). Items will process individually after scan check with sort out. Safety cleared pickup release notification to authorized carrier by or before this date or until further notice. Scancom servers remain offline at this time for vulnerability mitigation. Please await further information by manual transmission."
As of 9:18 am EST Representatives for US Customs Agents in charge have replied to our shipping agent. "Forecast updated for safety clearance process to begin as of 2022-01-15 05:00 LT (UTC -8) until further notice."
As of 8:07 am EST Representatives for US Customs Agents in charge have replied to our shipping agent. "Shipment is located within DHS CPS queue now awaiting safe transit clearance and release for carrier pickup."
Q: What does that mean?
A: The delayed shipment containing all affected customer battery shipments is currently in the possession of Dept of Homeland Security Customs and Border Patrol (US DHS CBP) agents at CPS (Central Processing Station). The shipment is now in the queue for final inspection and clearance to release all items to the licensed carrier FedEx who will make delivery to the recipient address on file.
This shipment contains batteries for all Triad customers currently affected by the delay caused by the Log4j server vulnerability and backlog of international shipments arriving during year end 2021.
DHS and FedEx servers still remain offline due to the Log4j vulnerability for international small parcels, air express and DG items which includes this group of Triad battery shipments.
Critical information regarding clearance, pickup notifications, and scan tracking data remains being generated manually by phone as the situation remains fluid while DHS and FedEx both work to bring their data centers back online. Their systems will not be brought back online until the complete Log4j system vulnerability has been patched.
The situation has been further complicated with severe weather, flight cancellations and a clearance agent and shipping carrier worker shortage due to mandatory 5 day sick leave time for vaccinated employees who have contracted break through cases of the Covid-19 omicron variant.
We will continue each day to monitor the progress surrounding these important Triad battery shipments and provide updates to this page for your reference. Please be sure to check junk mail and spam folders regularly for direct contact from FedEx shipping notifications once your item is cleared and in transit to your final destination.
Once the Log4j vulnerabilities for international and DG shipments is patched and updated throughout the FedEx Ship Manager system, notifications to recipients by email will process normally.
U.S. Customs is working beyond it's capacity to mitigate, patch and update their system wide vulnerability. They are also working beyond their capacity at this time to clear all all items for release to FedEx for final destination deliveries.
Information regarding shipments will be presented from US Customs agents to the bonded and licensed carrier FedEx only. FedEx will then notify recipients directly by email only. Outbound phone calls by FedEx to recipients for these tracking updates is temporarily suspended at this time.
We sincerely apologize to all our valued clients at Triad Electric Vehicles for this unforeseen and unprecedented delay. We are doing all we can to retrieve real time information and work with the agents in charge to clear these important shipments for all who have been affected at this time.
This is an unforeseen anomalous logistics complication and in no way is considered normal business operations. Battery shipments were in fact sent in advance of vehicle production to account for mandatory safe transit clearance times. Battery shipments have a normal clearance time of 7-10 days at Dept of Homeland Security Customs Border Control. It’s the law under IATA and IACO UN Regulations that they are the governmental regulatory agency who must clear this powerful class of battery to keep delivery drivers and consumers safe while items are transit according to DOT, FAA and US Labor laws.
FedEx is the licensed carrier who makes delivery of this class of safety cleared battery shipment. Once cleared by DHS then it gets released by them and FedEx is notified to collect for pickup, scan the item for tracking purposes and then the item immediately begins the final destination transit process which has a normal time frame of 3-4 days to the door of the receiver. Battery shipment total normal timeline is 12-14 business days, which is on target with the same arrival time of the primary Triad vehicle shipment.
Weather and other FedEx service disruptions
January 18, 2022
https://www.fedex.com/en-us/service-alerts.html Winter Storm Izzy is causing hazardous conditions across the U.S. Our top priority is the safety and well-being of our team members, as well as providing the highest level of service to our customers. Although contingency plans are in place, some service delays and disruptions can be anticipated. FedEx is committed to providing service to the best of our ability in areas affected by the winter storm and as local conditions allow. We will continue to monitor the situation to minimize the impact on service.
FedEx Operational Impacts
January 22, 2022
The explosive surge of the COVID-19 Omicron variant has caused a temporary shortage of available crew members and operational staff in the FedEx Express air network. The health and safety of our team members is our top priority. We are implementing contingency plans and adjusting operations to minimize delays while continuing to provide the best possible service to our customers during these difficult times. Volume currently moving through the network will be prioritized for processing. Deferred and Premium Domestic FedEx Express Freight and International Economy Freight pick-ups are currently suspended.
PLEASE READ THIS ENTIRE STATEMENT
US CYBERSECURITY & INFRASTRUCTURE SECURITY AGENCY (CISA) ISSUES EMERGENCY DIRECTIVE
You may already be aware of the fact that at this time a malicious, self-propagating program known as “Log4j” has affected millions of data servers across the USA encompassing all US Federal agencies including US Dept of Homeland Security Customs and Border Patrol, the US Postal Service, and huge data companies such as Amazon, Google, Oracle, Microsoft, FedEx, and thousands of other corporations, agencies and organizations.
To be clear, Triad Electric Vehicles data is unaffected is completely secured as we do not store data online or over cloud-based servers specifically out of extreme caution over privacy protections.
However, the Log4j data breach is a global disaster preventing FedEx from presenting tracking information on international shipments as they work to resolve the issue. Both United States Department of Homeland Security Customs and Border Patrol (US DHS CBP) and FedEx must complete their computer system mitigations before they can repair their data systems and resolve the issues to be able to provide electronic communications for tracking details on smaller international parcel sized shipments. Specifically, and generally this refers to Triad battery shipments at this time.
During this time, due to these extraordinary circumstances beyond our control, we unable to collect tracking details for international deliveries of small parcel packages due to Log4j vulnerabilities to USDHS CBP, FedEx, UPS and USPS data and tracking servers.
The items in transit to you remain in transit to you at this time using offline systems right now, however the tracking information and scans normally collected and stored on FedEx and USDHS CBP computer servers is unable to be presented by these companies and government agencies while they mitigate this historically massive vulnerability at a critical time of year for shipping.
Log4j is a program that was embedded into massive Apache servers by hackers and allows bad actors to access and take control of those computer systems remotely for complete and full control of these computer systems. Simply put, this is an unimaginably huge data breach where hackers have exploited a very small vulnerability in software running on millions of computer servers all throughout the entire US data infrastructure system.
There is no other way to say it. It’s been devastating to the U.S. supply chain during this critical time and companies and the U.S. federal government are doing all they can to mitigate the problems.
Tens of thousands of federal and private programmers are scrambling to solve the issue, however once they fix one problem it seems another one is created. These are really big data servers, shared systems owned by Amazon Web Services, Google Cloud, Alibaba Cloud, Oracle Cloud, IBM Cloud and others which are leased by the U.S. Federal Government and other U.S. big companies for data storage used for thousands of data points per second.
The DHL tracking system for Triad electric vehicle deliveries DOES NOT have any indications of being affected. Vehicle delivery tracking remains unaffected. FedEx is the carrier who holds the license for transporting high powered battery shipments after clearance from U.S. DHS CBP agents.
The Fedex and U.S. Customs systems used for battery safe transit clearance has been compromised by Log4j vulnerabilities. https://www.fedex.com/en-us/shipping/ship-manager/software.html#tab-4
Triad is calling DHS CBP representatives and FedEx on your behalf each day for any information they can provide, which may include details about their data security patches, offline updates about shipments arriving, updates to existing tracking numbers for server groups that have been repaired, additional service delays, or new tracking numbers for shipments in transit at this time that are in transit to final destinations. We will provide updates to this page as soon as FedEx and US DHS CBP Federal Agencies involved in safe transit clearance of batteries (and other small international air parcel packages) are able to respond to our daily requests.
As of January 7th, 2022 Triad managers communicating with representatives of DHS CBP have confirmed all items are in possession of the CBP Clearance Agents at LAX at this time; although DHS CBP representatives stopped short of providing an exact date they will complete their releases for pickup notices to FedEx in order to collect all items in their possession at this time.
DHS representatives however did provide an estimated date of release based on their work load at this time as January 14th, 2022 until further notice. If upon January 14th the releases are completed, FedEx will be notified to collect all items and scan them in locally to update the tracking details.
As of January 7th, 2022 FedEx trace managers also have stated the tracking numbers may change at the DHS pickup site upon release and the will make every attempt possible to update the receivers manually by email directly, although it is possible the packages may arrive without prior notice.
FedEx Ship Manager v.3509 Log4j – Service Bulletin Board Message
Log4j Update December 22, 2021
Due to the urgency of this issue, FedEx recommends upgrading immediately to version 3509 to consume the Apache Log4j 2.16.0 release to address the vulnerabilities for Java 8 users.
FedEx is actively assessing the situation related to the Log4j Remote Code Execution vulnerability.
FedEx Ship Manager™ Software (GSM)
Versions 13.62 and 14.56
Upgrade to version 14.57 upon release in January 2022
The above fixes will remediate the known issues associated with the Apache log4j vulnerability; any subsequent issues will be addressed as needed.
What is the risk?
According to Apache, some Log4j versions of this exploit are ranked 10 out of 10, with a score of 10 representing the most extreme vulnerability. This means an unauthenticated remote actor could exploit this vulnerability to take control of an affected system. To learn more about the risks and ranking click here.
We encourage customers to follow security best practices including those recommended by Apache (Apache Log4j Remote Code Execution), in addition to upgrading FedEx solutions as provided. Details are listed below:
FedEx Ship Manager® (FSM) Versions 340x and above
Upgrade to version 3509 to consume the Apache Log4j 2.16.0 release to address the vulnerability for Java 8 users.
For any additional related questions or the most updated information, customers should contact their Customer Technology representative.
FedEx Delays: Customers frustrated
Official Statement from FedEx "We understand our customers’ frustrations and apologize for any inconvenience as we accelerate contingencies to resolve delays caused by significant package volumes, IT vulnerabilities, and an ongoing industry labor shortage. FedEx is committed to providing service to the best of our ability as local conditions allow. We will continue to monitor these situations in an attempt minimize the impact on service."
Civilian personal computers are NOT LISTED AS BEING AFFECTED because this is a large-scale Apache data server vulnerability.
Government agencies and (including but not limited to) FedEx, USPS, and many other companies are taking their computer systems offline at this time as a precaution in response to the Log4j vulnerability. CISA says the temporary takedown reflects the extreme risk and urgency of patching the flaw.
Homeland Security secretary Alejandro Mayorkas said on Thursday that he is "extraordinarily concerned" about the vulnerability.
More Facts about Log4Shell (CVE-2021-44228) Vulnerability
- Originally reported on over 1 year ago with no investigation by any U.S. Government agency or mitigation by any U.S. company.
- Privately disclosed to the U.S. Apache Software Foundation by China’s Chen Zhaojun of the Alibaba's Cloud Security Team, again November 24th.
- U.S. Apache Software Foundation publicly disclosed to their U.S. clientele including the Federal Government on December 9th
- The issue was not addressed by any Federal or civilian organizations until December 17th, 2021
THE DHS CYBERSECURITY & INFRASTRUCTURE SECURITY AGENCY (CISA) ISSUES EMERGENCY DIRECTIVE REQUIRING FEDERAL AGENCIES TO MITIGATE APACHE LOG4J VULNERABILITIES
The Department of Homeland Security's Cybersecurity and Infrastructure Security Agency issued an emergency directive on Friday December 17th, 2021 that all federal and federal civilian agencies must assess their systems for mitigations related to the Log4j vulnerability by December 23, 2021.
CISA has determined that this vulnerability poses an unacceptable risk to Federal Civilian Executive Branch agencies and requires emergency action.
This emergency action is based on: (1) the current exploitation of these vulnerabilities by threat actors in external network environments, (2) the likelihood of the vulnerabilities being exploited, (3) the prevalence of the affected software in the federal enterprise, (4) the high potential for a compromise of agency information systems, and (5) the potential impact of a successful compromise.
Official USPS statement: “Computer networks are constantly under attack from criminals who try to exploit vulnerabilities to illegally obtain information. Similar to other companies, the Postal Service’s Information Security program and the Inspection Service uses industry best practices to constantly monitor our network for suspicious activity. Out of an abundance of caution, the Postal Service is further investigating to ensure that anyone who may have sought to access our systems inappropriately is pursued to the fullest extent of the law.”
As a valued customer of Triad Electric Vehicles, you can be assured that we will continue to contact our bonded shipping agents at origin and our FedEx agents here domestically on your behalf each day for manual updates.
Once the release is provided by DHS manually to FedEx as servers are offline, you will immediately be informed by email. FedEx managers have manually entered auto updates to receivers for their shipment(s) in transit at this time.
As of 12-22-21 U.S. Customs Agents have replied to our bonded shipping agents that it is expected for all battery and DG class items (dangerous goods) in process at this time are expected to be cleared by or before January 14th, 2022. At the time clearance is provided, notifications will be made to carriers to collect items from U.S. Customs locations for final delivery to recipients.
**NOTE: Because of the Log4J server issue affecting U.S. Customs and FedEx tracking scan databases, it is entirely possible that battery shipments may arrive unannounced without tracking because they are using offline systems at this time. FedEx is changing tracking numbers throughout the delivery process with handheld label printers at pickup and transfer stations without notifying us of tracking number changes. FedEx is currently utilizing smaller node intranet systems while their larger internet systems remain offline as they mitigate the Log4jShell vulnerability.
If items do arrive unannounced without tracking at this time, please let us know by sending a brief email with the final tracking number or a photo of the outer box packaging so we can cross reference this information with our FedEx battery shipping agents who are working hard during this time on your behalf to remain in contact with the U.S. Customs agents who clear all battery shipments for safe transit.
Federal Agents must by law clear all high powered battery shipments for safe transit before carriers can collect items for final delivery. More resources about the compounded delivery delays affecting FedEx and U.S. Customs clearance processes at this time is below.
Additional information affecting possible delays of battery shipments at this time.
Recent Updates about the container ship backlog at U.S ports
Inadequate CBP Staffing
DHS expands paid program to hackers to hunt down Apache vulnerability
December 27, 2021
Commercial airlines are commonly used by shipping companies to transport goods throughout the U.S supply chain.
Airlines are struggling with cancellations and delays over the holidays due to staffing shortages caused by the omicron variant.
Monday December 27 added more than 1,400 flights scrapped, with Chinese and U.S. destinations being the worst hit, the FlightAware data tracking website says.
US airlines say the disruption is due to crews testing positive or isolating.
More than 8,000 flights have been grounded over the long Christmas weekend that began on Friday.
December 30th, 2021
Additional flight cancellations continue to burden travel and cargo transit over New Year holiday weekend. https://www.nytimes.com/2021/12/30/business/air-travel-cancellations.html
January 4th, 2022
Log4j flaw attack levels remain high, Microsoft warns https://msrc-blog.microsoft.com/2021/12/11/microsofts-response-to-cve-2021-44228-apache-log4j2/
January 7th, 2022
Snowstorm Quickly Pushes U.S. Airline Cancellations Past 2,000
January 10th, 2022
FedEx Corporation FDX is taking a hit from Omicron-induced spike in coronavirus cases and adverse weather conditions across the United States. https://www.yahoo.com/now/fedex-fdx-warns-delays-amid-160004030.html