VMware Workstation 11 on Linux Mint 18

VMware Workstation 11 was released before Linux Mint 18 (and its parent, Ubuntu 16.04). Because the Linux kernel has changed, the Workstation installer fails to compile the vmmon and vmnet modules. Fixing this requires some minor patches. Once this is done, Workstation itself won’t start; fixing this requires a minor patch to a couple of shell scripts.

First, start a Terminal and become superuser by using sudo su. Then run the module compilation tool to see the errors:

vmware-modconfig --console --install-all

On my system, the errors looked like this in vmmon:

error: void value not ignored as it ought to be
retval = misc_deregister(&linuxState.misc);

and this in vmnet:

error: too few arguments to function sk_alloc

You can fix these errors by patching the modules source. First, move to the /usr/lib/vmware/modules/source directory. Then extract the vmmon and vmnet module sources:

tar xvf vmmon.tar
tar xvf vmnet.tar

Then apply the following patch to vmmon:

--- vmmon-only/linux/driver.c	2015-03-17 22:25:55.000000000 -0400
+++ vmmon-only/linux/driver.c	2017-03-01 06:33:08.000000000 -0500
@@ -281,9 +281,7 @@
 #ifdef VMX86_DEVEL
    unregister_chrdev(linuxState.major, linuxState.deviceName);
 #else
-   if (misc_deregister(&linuxState.misc)) {
-      Warning("Module %s: error unregistering\n", linuxState.deviceName);
-   }
+   misc_deregister(&linuxState.misc);
 #endif

    Log("Module %s: unloaded\n", linuxState.deviceName);

and apply the following patch to vmnet:

--- vmnet-only/vmnetInt.h	2015-03-17 22:25:55.000000000 -0400
+++ vmnet-only/vmnetInt.h	2017-03-01 06:40:00.000000000 -0500
@@ -78,9 +78,14 @@
 
 extern struct proto vmnet_proto;
 #ifdef VMW_NETDEV_HAS_NET
+#if LINUX_VERSION_CODE < KERNEL_VERSION(4,2,0)
 #   define compat_sk_alloc(_bri, _pri) sk_alloc(&init_net, \
						 PF_NETLINK, _pri, &vmnet_proto)
 #else
+#   define compat_sk_alloc(_bri, _pri) sk_alloc(&init_net, \
+                                                PF_NETLINK, _pri, &vmnet_proto, 1)
+#endif
+#else
 #   define compat_sk_alloc(_bri, _pri) sk_alloc(PF_NETLINK, _pri, &vmnet_proto, 1)
 #endif

After applying these patches, replace the source tar files:

mv vmmon.tar vmmon.tar.old
tar cvf vmmon.tar vmmon-only
rm -rf vmmon-only
mv vmnet.tar vmnet.tar.old
tar cvf vmnet.tar vmnet-only
rm -rf vmnet-only

Then rerun the module compilation tool:

vmware-modconfig --console --install-all

If the compilation succeeds, you can now patch the Workstation run scripts. Add the following line to the scripts /usr/bin/vmware and /usr/bin/vmware-netcfg:

export LD_LIBRARY_PATH=/usr/lib/vmware/lib/libglibmm-2.4.so.1/:$LD_LIBRARY_PATH

Try running /usr/bin/vmware from a Terminal session to verify that it works, or to see any errors if it doesn’t work.

Finally, you might have to fix a menu problem in Workstation. On my installation of Mint 18, Workstation didn’t show all of its menus. Only the File menu was visible, but the others could be made to appear by hovering over the spot where they were supposed to be. This can be fixed by adding the following line to /etc/vmware/boostrap:

export VMWARE_USE_SHIPPED_GTK="yes"

This fix is also required for Workstation 12.5.