Tuesday, March 23, 2021

Lenovo Thinkpad TB3 Dock Saga

Around fall 2020, my old desktop PC was on its last leg (it was running Intel Core Quad 9300 - so it was super old). I decided to get a reasonable powerful laptop with a dock that can drive two 2K (2560x1440) and one 1920x1200 monitors. So I purchased a X1 Carbon Extreme Gen 2 (running i7-9750H, 32 GB RAM) and Thinkpad Thunderbolt 3 Workstation Dock Gen 2 (40AN) - and I also purchased the Premier Support. But soon, it was apparent to me something was wrong.

I always use my laptop with the lid close, connected to the dock with the provided custom cable (which provide TB3 connection and power), with everything else connected to the dock (3 monitors, USBs, network).

The problem starts with intermittent screen flickering - which I initially dismissed as just small annoyance or maybe lose connection/cable. But it got to the point where it was annoying enough for me and I contacted Lenovo Premier Support. I started my case on January 4th and after about 11 weeks - it is now closed (but not resolved).

Lenovo and their team has been very professional in responding to the issue. Every call I made to the Premier Support team or email exchanges have been helful and courteous. They deserve a lot of credit in investigating the issue and trying to find a good resolution with me.

So what is the main issue?

In short, to the user (me), the external monitors will blink (goes blank and comes back within a second or less) and some times will go blank for a period of time (ranging from two seconds to several minutes, or in some minor instances actually never come back). After about 8 weeks, Lenovo admitted that there is a bug in their dock - or to be precise in the chip (from Synaptics) they are using in the dock. The bug is related to HDCP - which is digital copy protection to prevent copying of digital audio and video content (i.e. during streaming). Since this chip is being used in all their TB3 docks (and Lenovo said also being used in other brands like DELL and HP docks) and it cannot be fixed via firmware update, so Lenovo said they do not have a workable solution for this issue (for me and other dock owners). I ended up trying to get a refund for my dock.

The issue was happening often enough for me to the point I was able to create reproducible steps (and the issue will happen 90% or more when followed):
  • Make sure laptop is connected to dock using the Lenovo provided TB3 cable
  • Make sure laptop lid is closed
  • Make sure external monitor(s) are connected through the dock
  • Login to Windows
  • Open Edge (or Chrome) in on of the external monitor
  • Go to https://www.netflix.com using that browser and login
  • Wait for the main Netflix page and one of the trailer on the banner to play in the background (usually screen starts to flicker or gone completely black at this point)
  • Click on one of the movies (any movies) and play it - then click the full-screen icon on the bottom right (sometime the flicker/black screen will happen here)
  • To repeat - you will need to close all browser windows and redo from going to Netflix website step
It also happened with https://eshop.macsales.com:
  • Laptop lid is closed, laptop is connected to dock via TB3
  • Restart laptop, login
  • Go to https://eshop.macsales.com/shop/docks (using any browser – but I was using Edge Chromium)
  • Scroll up and down on the website using the mouse wheel fast
  • Click on a link on the top menu (or go to a different page on the site) and scroll up and down
  • You will see one of the screen blink or blank (from quick blink to sometimes blank for several seconds)
If you are interested in the chronological timeline of the saga, here it is:
  • Jan 4 - Call Premier Support, talked about the issue and technician diagnosed the issue maybe with the defective dock. Lenovo sent replacement dock (overnight).
  • Jan 6 - After trying the new dock, issue still persist, call Premier Support again. Technician did some digging and remoted into my machine and eventually concluded that maybe my main laptop board is defective (not the dock - since it is newly replaced) and scheduled a technician to come to my house to do a main board replacement on the laptop
  • Jan 8 - Technician showed up at my house and replace my X1 Extreme main board. Issue still persist and technician told me that he could not fix it since it is most likely not a laptop main board issue and recommended me to call Premier Support again
  • Jan 11 - Call Premier Support again. This time I recorded some videos of the issue happening. Technician was stumped and tried to escalate to his technical lead, but the lead has left and said he will follow up.
  • Jan 12 - Since I did not get any follow up, I called Premier Support again, but this time I was able to talk to a technical lead. The technical lead remoted into my machine and did some data collection and diagnostic. I also sent the videos of the issue. The lead said he would need to talk to an engineer about the issue.
  • Jan 19 - The technical lead emailed and sent me some information from the engineer and seeking more information about the dock (S/N, etc) and I sent the picture of the bottom of the dock which has all the information about the dock.
  • Jan 21 - A Lenovo System Support Engineer (Thinkpad Level 2) reached out to me confirming that he was able to replicate the issue. He asked me about my Windows update/version and whether I tried it with YouTube or other streaming service. I replied saying that I could get the issue to happen with DisneyPlus (I also sent video of it), but not with YouTube or CBS Plus.
  • Jan 25 - The System Support Engineer wanted me to test with FireFox - which I did and the issue did not happen. But the issue happened when using Netflix app (instead of the browser).
  • Jan 27 - The System Support Engineer escalated the issue to Product Engineering team
  • Feb 1 - The Product Engineering team escalated the issue to the dock engineer
  • Feb 9 - The dock engineer was able to replicate the issue in the lab and initial analysis seems to point to the HDCP as the cause of the issues
  • Feb 16 - Lenovo says they were working with Synaptics (the chip's vendor) to try to come up with a solution
  • Mar 1 - Lenovo minimized the issue and practically denied that this is an issue out of the ordinary - which I guess was probably because the engineer that replied was not the System Support Engineer that I have been working with, so he probably has not seen any video recordings I have done etc - so I responded to him by letting him know that his assessment was not my exprience and I sent him the videos I recorded of the issue.
  • Mar 9 - The System Support Engineer updated me saying that Lenovo and Synaptics were able to replicate the issue and confirmed that issue was cause by HDCP calls.
  • Mar 12 - After some emails back and forth (basically I was asking for clarification to help me understand the issue better), then Lenovo confirmed that the issue cannot be fixed in the current dock with the chipset.
  • Mar 12 - I then responded by asking about the next step - should I get a refund? Or a replacement with the new 2021 dock? etc.
  • Mar 12 - After I sent my email, a different person from Lenovo replied (this person has been CC'd in most of the emails) minimizing the issue again, saying that it should only blink one second and no more - as well as saying that this is an "industrial wide limitation" and exhibited in other brands (HP, DELL). I assume that this person has probably not viewed my videos and I replied asking him that and also stating that my experience was certainly not as he described (blink 1 seconds or less and no more).
  • Between Mar 12 - Mar 16, this person from Lenovo corresponded with me and basically doing some troubleshooting and logging. He would sent me a logger to execute and then I would run it and repeated my replication steps to reproduce the issue. Then I would send him logs. He would then send a debugging patch to execute and repeat the logging stuff, etc - repeated several times.
  • Mar 17 - The person I have been corresponding with then concluded the same conclusion - that the issue is related to HDCP authentication and cannot be fixed for the current dock with the current chip.
  • Mar 17 - On that same day, Lenovo opened up a Customer Satisfaction team case to help me so I can get a refund on my dock.
  • Mar 22 - A member of Customer Satisfaction team reached out to me to help me with the process of getting a refund for my dock.
For the video recordings of the issue - they are available here and here.

2 comments:

Anders Heick said...

Thanks Johannes.

Great details and good explanation. I am curious if you found an alternative solution that will allow you to run with the desired monitor setup?

/Anders

Johannes Setiabudi said...

I actually do not - I ended up returning the dock and use direct HDMI and USB-C/TB3 dongle (that has dual HDMI) to connect to my monitors. I have multiple cables (instead of just 1 cable) now going to my laptop (power, HDMI, and TB3 dongle) - but at least it is very reliable now.