Transferring files between remote devices and systems is a critical task in today’s digital world. Although FTPS and SFTP are widely used for remote file transfer, they are often slow when it comes to handling small files (10 KB and less). The more small files there are to transfer, the more impact this issue has. With large volumes of data, backup time can be heavily penalized.
ByteSync drastically reduces data transfer times, being at least 10x faster than FTPS and SFTP for synchronizing small files. In this post, I first present a comparison of small file transfer performance between FTPS, SFTP and ByteSync. I then discuss why FTPS and SFTP protocols are slow before explaining why ByteSync is so fast at transferring small files.

Small File Transfer Performance Comparison: FTPS, SFTP vs ByteSync

To highlight the differences in transfer speed of small files, I performed comparative tests remotely between the FTPS and SFTP protocols on the one hand, and the ByteSync software on the other. I did not retain the FTP protocol for these comparisons because, although lighter and certainly faster than FTPS, it has the major disadvantage of transferring data in clear text over the network, which is prohibitive in terms of security for backup or remote synchronization via the Internet.

To perform the tests, I used a Windows machine located in my local network and a Linux virtual machine hosted by OVH (Virtual Private Server). They are both mid-range and are connected to the Internet with fiber optic connections. I used the following software configurations:

• For FTPS tests, the Windows machine uses the WinSCP client and the Linux virtual machine uses the vsftpd server, with a standard configuration.
• For SFTP tests, the Windows machine uses the WinSCP client and the Linux machine uses the default SSH server.
• For the ByteSync tests, ByteSync is deployed in portable mode on the Windows machine and on the Linux machine.

To take relevant measurements, I transferred small files from the Windows machine to the Linux machine in two test sets. The first had 10,000 10k files, the second 10,000 1k files.
I also took into account that increasing the number of parallel FTPS and SFTP connections will decrease the overall backup time. I used the usual approach of existing backup synchronization software, which is to use 1 or 2 simultaneous connections to send the data.
The two tables below show the best results obtained after several tests.

Sending 10 000 files of 10 kb (102 400 000 bytes):

Sending 10 000 files of 1 kb (10 240 000 bytes) :

Before commenting on these results, I want to emphasize that they should not be taken as a universal truth. While I have obtained similar values on several occasions, these are dependent on the hardware, software, and network configurations implemented and it is likely that performance will vary in other test environments.
Also, my web searches generally indicate that FTPS is faster than SFTP (for example here, and here). However, in the results presented in this post, SFTP is clearly 3 to 4 times faster than FTPS. Since both servers are configured in a standard way, I am not able to explain why SFTP has a clear performance advantage.

In the context of these tests, we find that:

• ByteSync is much faster than the FTPS protocol: the ratio goes from 50 to 100 in favor of ByteSync!
• ByteSync is much faster than SFTP: the ratio goes from 15 to 30 in favor of ByteSync!
• ByteSync is 20% faster at handling 1k files than 10k files, where FTPS transfer is slower (?!?) and SFTP transfer is only 7% faster. Yet there is 10 times less data to transmit!

You can see for yourself the difference in performance in the video below. Warning: ByteSync’s small file transfer speed has doubled since this video was recorded.

Why FTPS and SFTP are slow on small files

File Transfer Protocol Secure (FTPS) and Secure/SSH File Transfer Protocol (SFTP) are commonly used to transfer files over the Internet. Although different, these two protocols have many similarities. They are based on the Transmission Control Protocol (TCP), are designed to provide secure communication between the client and the server, and offer similar operating logic.
Unfortunately, the FTPS and FTPS protocols are not very efficient for handling small files. As we have seen above, as the file size decreases, the overall transfer speed drops. There are three major reasons for this ineffectiveness, all of which have cumulative negative consequences.

Negotiation process

One of the main causes of this slowness is the negotiation process that is required for each file transfer. This negotiation involves several information exchanges between the client and the server. The process can be summarized as follows:

1. The client announces that he wants to send a file.
2. The server authorizes the client’s request.
3. The client sends the data.
4. The server acknowledges receipt of the data, closes the file descriptor and updates the file dates.

Since this negotiation process is repeated for each file transfer, steps 1, 2 and 4 are systematic and have the same duration, regardless of the file size. Therefore, the smaller the files, the shorter the step 3 and the greater the impact of steps 1, 2 and 4 on the total transfer time.

Network latency

In addition, network latency can slow down the round trips between the client and the server and have a significant impact on the performance of FTPS and SFTP protocols. Network latency is the time it takes for data to travel from one point to another on the network, and can be influenced by various factors such as the geographic distance between the two points, the quality of the connection, network congestion, etc.
The higher the latency, the more time there is between the moment the client sends a command and the moment the server receives it, and vice versa. This causes delays in the server’s response to client commands and slows down steps 1, 2 and 4 of the negotiation process.
Whether in FTPS or SFTP, during step 3, the client sends data using the TCP protocol. Thus, the data is transmitted in consecutive segments without waiting for the server to acknowledge. The server will be able to acknowledge the reception of several data segments at once. As a result, network latency has less of an impact on stage 3 than on the other stages, as parties less often need to wait for the other party’s response to perform their processing. This operation favors the overall transfer speed of large files.

For more information on how the TCP protocol works, you can consult the article https://www.ionos.com/digitalguide/server/know-how/introduction-to-tcp/.

TCP overhead

The TCP header causes an overhead that is proportionally much higher on very small files. Present at the beginning of each transferred segment, the header includes the information necessary for the transmission of data on the network. The size of the header can vary, but in most cases it uses 20 bytes in IPv4 and 40 bytes in IPv6 (it can be longer, up to 40 additional bytes, if it contains additional TCP options).

The impact of the TCP overhead depends on the size of the packets in transit. Considering a MTU (Maximum Transmission Unit) of 1,500 bytes, which is a common value on the Internet :

• If the length of the file to be transferred is 1000 bytes, the TCP header in IPv4 (20 bytes) represents a 2% overhead, and the TCP header in IPv6 (40 bytes) represents a 4% overhead.
• If the length of the file to be transferred is 100 bytes, the TCP header in IPv4 represents a 20% overhead, and the TCP header in IPv6 represents a 40% overhead.
• If the length of the file to be transferred is 10 bytes, the TCP header in IPv4 represents a 200% overhead, and the TCP header in IPv6 represents a 400% overhead.
• When the file is larger and has to be split into several packets, the average TCP header overhead remains proportionally low compared to the file size.

For more detailed case examples, please see https://packetpushers.net/tcp-over-ip-bandwidth-overhead/.

Why ByteSync is much faster than FTPS and SFTP for transferring small files

ByteSync is a high performance file synchronization solution. To transfer files, the solution uses HTTPS (Hypertext Transfer Protocol Secure) and passes them temporarily through Azure servers. For security, data is encrypted before transmission in AES-256 as part of the end-to-end encryption (E2EE) specific to each synchronization session. Initially designed for the transfer of Web content, and now massively used in APIs and Cloud exchanges, the HTTPS protocol works differently from the FTPS and SFTP protocols, but like them, is based on the TCP protocol.

Thus, the issues related to network latency and TCP header, which we have seen above, also apply to small file transfers with the HTTPS protocol. In addition, because the sending ByteSync client first uploads the data to the Azure servers before the receiving ByteSync client downloads it, new latency is introduced with each transfer. In initial measurements with ByteSync, latency related to buffering on Azure servers was much higher than slowness related to the FTPS and SFTP protocol negotiation processes.

In order to optimize the transfer speed of ByteSync and achieve the excellent results presented above, two complementary software systems have been developed and integrated into the solution.

Temporary transfer archive

Instead of transferring small files one by one, ByteSync adds them to a temporary transfer archive. When this archive contains enough files, it is encrypted with AES-256, uploaded to the Azure servers, and then downloaded from the Azure servers by the recipient client. It is then extracted by the recipient client and finally, the creation and last modification dates of the files are restored.
Thus, for 10,000 files of 10 kb, the data is sent in the form of a dozen archives. We then go from 10,000 uploads and downloads to 12 uploads and downloads.
Compared to the slowness of both the FTPS and SFTP negotiation processes, the TCP overhead, and the buffering on the Azure server, it is obvious that the time to prepare and retrieve these temporary archives is negligible.

Operations parallelization and resource management

As the operations of creating, transferring and extracting the temporary transfer archives each take a long time (several seconds at least), a significant optimization work has been done in terms of parallelization of the actions and memory management. Everything is done to ensure that the actions are carried out in parallel without overloading the use of the resources available on the machines.

Thus, while a temporary transfer archive is being prepared on the sending client, a previously created archive is transferred while an older one is decompressed by the recipient client.

ByteSync goes even further

As mentioned earlier, ByteSync uses Azure servers to act as a buffer in the cloud between the sending and recipient client. This is the basic operation for a session with 2 remote clients.
However, ByteSync allows up to 5 remote clients per session. And in the case where the same data is sent to multiple clients, the sending client sends it to the Azure servers all at once and then each recipient client downloads it in parallel. This is more efficient than synchronization with FTPS and SFTP, in which the same data must be sent by the sender to each recipient.
Thus, if we apply the examples of the above comparisons to a case where 1 client sends the same data to 4 remote clients, the overall transfer time of ByteSync remains roughly the same, where the overall transfer times in FTPS and SFTP are multiplied by 4!

Furthermore, ByteSync does not reserve the temporary transfer archive system solely for the initial synchronization of small files. These archives are also used for transferring differences between files, which are stored in deltas files, if these deltas files are small enough. Thus, even in the case of massive resynchronization where the differences between files are few, ByteSync’s performance is optimal.

ByteSync’s difference handling and delta compression are not covered in this post, but I wanted to point out that even in this case, everything is done for optimal performance!

Conclusion

FTPS and SFTP protocols can be slow when it comes to synchronizing or backing up small files. This is because their negotiation processes make the transfer of small files more cumbersome, and the TCP segment headers affect the exchange of very small files more negatively. In addition, the impact of network latency, which slows down the flow of information back and forth between client and server, is proportionally greater on small files.
Moreover, this transfer speed issue also affects many of the backup and synchronization software that operate file by file. The more small files there are to transmit, the more it is felt.

However, the slow transfer of small files is not inevitable. ByteSync, which also relies on TCP and uses Azure servers as a buffer, is subject to similar constraints. To overcome these difficulties, the software uses a temporary transfer archive into which the files to be transferred are copied, and it uses advanced parallelization processes to avoid downtime.

For example, ByteSync data synchronization software is 10x faster than FTPS and SFTP protocols for exchanging small files. This performance makes it the ideal choice for anyone who frequently transfers small files and needs a truly efficient solution. If you are looking for a fast, secure and easy-to-use file transfer solution, check out ByteSync!