In Video-based Point Cloud Compression (V-PCC), 2D videos to be encoded are generated by 3D point cloud projection, and compressed by High Efficiency Video Coding (HEVC). In the process of 2D video compression, the best mode of Coding Unit (CU) is searched by brute-force strategy, which greatly increases the complexity of the encoding process. To address this issue, we first propose a simple and effective Portable Perceptron Network (PPN)-based fast mode decision method for V-PCC under Random Access (RA) configuration. Second, we extract seven simple hand-extracted features for input into the PPN network. Third, we design an adaptive loss function, which can calculate the loss by allocating different weights according to different Rate-Distortion (RD) costs, to train our PPN network. Finally, experimental results show that the proposed method can save encoding complexity of 43.13% with almost no encoding efficiency loss under RA configuration, which is superior to the state-of-the-art methods. The source code is available at https://github.com/Mesks/PPNforV-PCC.