The TPLink Archer C80 is a powerful, cheap and easy to setup wifi router but unfortunately it lacks the range extender functionality. Of course this isn’t a real hardware limitation but only a software (firmware) deactivated feature. Luckily there is a way around it.
What I want to accomplish is to connect the TPLink to another WIFI network so that its 4 Gigabyte Ethernet ports would be able to surf on it and to the internet connected to its wifi router. This is usually done by a simple range extender setup/ wizard on other routers. Unfortunately this mode is not present in the C80 configuration Advanced > Operation Mode (probably removed because there is a reference for it in the help – search for range extender) but it can be somewhat obtained anyway.
This method has been tested on a router with hardware version 2.20 and firmware Archer C80(EU)_V2.20_1.13.2 Build 230824 , which is the newest one as of April 2024, and it works without any drawbacks (even when the router is restarted or shutdown) apart from the fact that the router itself will not be accessible anymore (you must reset it if you need to modify the configuration – not a big deal after all).
In my case I wanted to use my internet provider’s wifi router with IP 192.168.1.1 and DHCP, NAT and all the usual services enabled.
Connect to the TPLink via an ethernet cable (don’t use the WAN port) and leave the initial wizard with the default choices (the Tether mobile application is not helpful for this setup, don’t use it).
Use the Default Router Mode (not the Access Point Mode);
Leave the Advanced > Network > Internet settings unmodified (Dynamic IP) but disable DoH, NAT and NAT Boost.
Disable Network > Wireless > WPS
In Advanced > Wireless > WDS, following the manual, setup a WDS bridge to the WIFI you want to connect to (choose the right antenna for your case, you cannot use both).
Disable the DHCP service on the TPLink (Advanced > Network > DHCP Server);
In the Advanced > Network > LAN set the IP address of the TPLink router to the same address of the WIFI router you want to connect to (192.168.1.1 in my case). This is because the TPLink firmware prevents you to change the routing table to route the traffic to the WIFI network connected via WDS but using the same address of the target network for the TPLink itself will set the routing table the way it should be. It won’t work now of course because the packets sent to the external router will be intercepted by the TPLink (all the other WIFI connected devices should be reachable though).
To bypass the problem about the TPLink intercepting the packets sent to the target WIFI router change the LAN IP address of the TPLink to another IP (I used 192.168.0.1): the trick works because this change will not modify the routing table (possibly a lucky bug) and so all your packets will finally reach the intended target of your external WIFI router and through it to the internet!
I get it: JPEG-XL is a wonder (and I like it very much)! It’s good, it’s fast, it’s progressive, it supports enormous image sizes and can losslessly encode older JPEG images: it has all the rights to be the king of digital photographic environments. AVIF is quite there too (but clearly inferior as a pure image format) but the elephant in the room, something that was not considered in the many reviews I have read so far, is that AVIF is the AV1 keyframe image format! This means that in a few years all graphic adapters on every device will be able to decode it in hardware using the same circuitry already present to decode AV1! This will never be available for JPEG-XL that would require a complete new dedicated hardware! And that’s the end of it. CPUs nowadays are going toward specialized hardware accelerators and this is just another example of it.
The letter should be concise and to the point because nobody likes to read a long and confused text. I hope this model meets the premise:
Dear On.le Xxx Yyy, I am an independent software developer and I work with and collaborate on Open Source projects. I am very concerned about the proposed Cyber Resilience Act because I think it will hurt collaborative software development at its core without any real impact on security.
The Proposal wrongly assumes that security could be assessed and certified but unfortunately that is not the case: software security can be improved by a methodology (it’s a characteristic of a quality process) but cannot be assessed or measured in any way. It’s technically hard to even pinpoint security responsibilities that might depend on the interaction of many different parts. This puts the law on a very unstable ground and the fear of being randomly fined represents an unacceptable risk to all independent developers, collaborators and users of Open Source projects and tools and small software houses in general.
The principle of proportionality doesn’t apply to security: even a very small contribution or code change can subtly and unknowingly introduce severe bugs and security issues so to adapt the requirements to the scope of a contribution will not be effective in fighting bugs and improving security at all but will pose an unacceptable and almost random risk to any contributor effectively discouraging participation and the very existence of Open Source projects.
Libraries are pieces of code that can be included in other projects. The author of a library can not know how, when and where its library will be used and can not take appropriate security actions based on the target environment (even if that would make any sense at all). To review every single code line of every single library included in a project (they might be easily thousands) can not be done within the limit of a reasonable budget even once or twice in a project life, let alone at every single release.
This law will be disruptive in the market by requiring assurances and legal help, adding a considerable burden over independent developers and software houses without affecting security at all. Limiting the requirements of the law to major software only would not help either because it just takes a single small application (tool, utility, game) to open a system backdoor.
While today developers share bugs and security issues eagerly and promptly (their reputation being on the line) the sanctions of this Proposal might discourage that positive attitude effectively worsening the general security approach rather than improving it.
I have written a blog post with a detailed critique of the law that might interest you:
The Proposal takes care of the software cybersecurity requirements by establishing a chain of responsibility, a specific certification and severe penalties.
The cyber security problem is approached in a similar way as GDPR approach privacy by establishing a chain of responsibility. But those two are fundamentally different things: while privacy is an established citizen right and can be promptly and definitely assessed by observing the shared personal data, security is a non measurable quality. A manufacturer trust worth is determined by its security history (something that can be evaluated only in time) and the market automatically adapts its choices to its security needs. How can security be assessed otherwise? There is no way to assure that a code is error free!
Determining the responsibility of a security issue might be technically challenging as every software produced nowadays comprises, in its development cycle, code and modules written by thousands of other subjects (often impossible to track). Bugs often comes out from unexpected interactions between unrelated codes. What happens if a compiler introduces a security issue into my code? What happens if that compiler is free software (not covered by the legislation)?
While this Proposal effectiveness in fighting cyber crime and security issues is debatable its impact on the Open Source community and the smaller software producers in general is quite certain and destructive because the risk of being randomly fined is unacceptable.
There is an unbalance in the law based on the needed proportionality of the effort: a small application (i.e. a little mobile game) has less security requirements than an security related one (i.e. a VPN manager). But small, apparently innocuous, applications are subject to security attacks probably even more than expensive products with equal or even greater impact on systems (i.e. a backdoor) so while we spend time and effort to enforce protection on big applications, the smaller one will still be there to create troubles. Except for the case where this legislation would put all small producers out of the market directly.
The only reasonable approach to security is by enforcing guidelines like those used by the Quality Control Systems because security is in fact a characteristic of the quality process. But those guidelines are already in place!
Instead of promoting security this Proposal will just create a new market for software agencies and developer assurances against security risks.
This Regulation ensures a high level of cybersecurity of products with digital elements. It does not regulate services, such as Software-as-a-Service (SaaS)[…] All entities providing cloud computing services in the Union that meet or exceed the threshold for medium-sized enterprises fall in the scope of that Directive.
The intent seems to steer the market from the (potentially insecure) installable software in favor of cloud services. Even though this might be the right direction, there are conspicuous exceptions and important and subjective privacy concerns, the scope of the law should not be to influence the market. Moreover the security of a SAAS infrastructure might reveal itself even more critical than that of an installed software so why should it be excluded by the security requirements of this Proposal (maybe because this is the big companies main business now)?
10
In order not to hamper innovation or research, free and open-source software developed or supplied outside the course of a commercial activity should not be covered by this Regulation. This is in particular the case for software, including its source code and modified versions, that is openly shared and freely accessible, usable, modifiable and redistributable. In the context of software, a commercial activity might be characterized not only by charging a price for a product, but also by charging a price for technical support services, by providing a software platform through which the manufacturer monetises other services, or by the use of personal data for reasons other than exclusively for improving the security, compatibility or interoperability of the software.
This point excludes only nonprofit Open Source projects which are a distinct minority. Even those not monetized directly might be eventually forked or included in some other monetized project by someone else (unknowingly to the original author). Consider that even offering services on a product (even only formation) constitutes a form of monetization. This means that either developers will be responsible of every code published or contributed to or the final distributor of the product has to constantly assess and certify the security of every line of code present. In both cases collaborative software productions will become an unaffordable risk.
21
In order to ensure that manufacturers can release software for testing purposes before subjecting their products to conformity assessment, Member States should not prevent the making available of unfinished software, such as alpha versions, beta versions or release candidates, as long as the version is only made available for the time necessary to test it and gather feedback. Manufacturers should ensure that software made available under these conditions is only released following a risk assessment and that it complies to the extent possible with the security requirements relating to the properties of products with digital elements imposed by this Regulation. Manufacturers should also implement the vulnerability handling requirements to the extent possible. Manufacturers should not force users to upgrade to versions only released for testing purposes.
Beta versions, rolling versions, incomplete versions and such are not excluded. It might be argued if Continuous Delivery might fall in this category as well.
22
In order to ensure that products with digital elements, when placed on the market, do not pose cybersecurity risks to persons and organisations, essential requirements should be set out for such products. When the products are subsequently modified, by physical or digital means, in a way that is not foreseen by the manufacturer and that may imply that they no longer meet the relevant essential requirements, the modification should be considered as substantial. For example, software updates or repairs could be assimilated to maintenance operations provided that they do not modify a product already placed on the market in such a way that compliance with the applicable requirements may be affected, or that the intended use for which the product has been assessed may be changed. As is the case for physical repairs or modifications, a product with digital elements should be considered as substantially modified by a software change where the software update modifies the original intended functions, type or performance of the product and these changes were not foreseen in the initial risk assessment, or the nature of the hazard has changed or the level of risk has increased because of the software update.
Security errors can be introduced (or triggered) by very small and simple changes (which are often less cared about and less tested). An updated dependency is a substantial modification? It’s a very small code change but might imply a huge risk. This requirement forbids the use of the Continuous Delivery process and the iterative Agile Management which is the main way of producing software today. Should every build of the project be the subject of a security assessment? The legislator knows that there could be several builds released every single day (i.e. Firefox nightly builds)?
25
Products with digital elements should be considered critical if the negative impact of the exploitation of potential cybersecurity vulnerabilities in the product can be severe due to, amongst others, the cybersecurity-related functionality, or the intended use. In particular, vulnerabilities in products with digital elements that have a cybersecurity related functionality, such as secure elements, can lead to a propagation of security issues throughout the supply chain. The severity of the impact of a cybersecurity incident may also increase when taking into account the intended use of the product, such as in an industrial setting or in the context of an essential entity of the type referred to in Annex [Annex I] to Directive [Directive XXX/ XXXX (NIS2)], or for the performance of critical or sensitive functions, such as processing of personal data.
An apparently innocuous software module might be used in a critical project. How can a developer establish what will be the use and the security requirements of its module beforehand? He might never even know.
44
[…] The conformity assessment procedures should examine and verify both product and process-related requirements covering the whole life cycle of products with digital elements, including planning, design, development or production, testing and maintenance of the product.
Open Source is excluded from the provisions of the law only if non profit which is a tiny fraction of the Open Source ecosystem. All other projects will have to simply shut down as it is impossible to comply with the requirements of this law by a collaborative project.
Art 10.5
The manufacturer shall systematically document, in a manner that is proportionate to the nature and the cybersecurity risks, relevant cybersecurity aspects concerning the product with digital elements, including vulnerabilities they become aware of and any relevant information provided by third parties, and, where applicable, update the risk assessment of the product.
The risk of a code can not be estimated beforehand, because it is determined by how and where this code is used. How a software module (i.e. a library) is used is completely outside the knowledge of its author(s).
Art 10.6
When placing a product with digital elements on the market, and for the expected product lifetime or for a period of five years from the placing of the product on the market, whichever is shorter, manufacturers shall ensure that vulnerabilities of that product are handled effectively and in accordance with the essential requirements set out in Section 2 of Annex I. […]
Art 11.1
The manufacturer shall, without undue delay and in any event within 24 hours of becoming aware of it, notify to ENISA any actively exploited vulnerability contained in the product with digital elements.[…]
Art 11.2
The manufacturer shall, without undue delay and in any event within 24 hours of becoming aware of it, notify to ENISA any incident having impact on the security of the product with digital elements.[…]
These obligations necessarily imply the manufacturer to be the subject of a potentially very damaging inspection and potentially a fine after a security issue is reported. Wouldn’t that stop or at least discourage manufacturers to report security issues? Wouldn’t that be against the basic intent of this Proposal?
Art 13.1
Importers shall only place on the market products with digital elements that comply with the essential requirements […]
A developer/company should asses every single software module used in its final product (and possibly during its development). This is simply outright absurd! A simple nodejs project might include more than a thousand modules developed by many different developers outside the EU!
Art 15
An importer or distributor shall be considered a manufacturer for the purposes of this Regulation and shall be subject to the obligations of the manufacturer set out in Articles 10 and 11(1), (2), (4) and (7) where that importer or distributor places a product with digital elements on the market under his or her name or trademark or carries out a substantial modification of the product with digital elements already placed on the market.
Art 53, Penalties
3. The non-compliance with the essential cybersecurity requirements laid down in Annex I and the obligations set out in Articles 10 and 11 shall be subject to administrative fines of up to 15 000 000 EUR or, if the offender is an undertaking, up to 2.5 % of the its total worldwide annual turnover for the preceding financial year, whichever is higher.
4. The non-compliance with any other obligations under this Regulation shall be subject to administrative fines of up to 10 000 000 EUR or, if the offender is an undertaking, up to 2 % of its total worldwide annual turnover for the preceding financial year, whichever is higher.
5. The supply of incorrect, incomplete or misleading information to notified bodies and market surveillance authorities in reply to a request shall be subject to administrative fines of up to 5 000 000 EUR or, if the offender is an undertaking, up to 1 % of its total worldwide annual turnover for the preceding financial year, whichever is higher.
Considerations
There are several reasons why this proposal is dangerous to the actual software market and wrong in its intents to fight cyber-crime:
Security issues are mostly due to bugs and there is no way to assure that a software is error-free (let alone by means of a certificate!). Incidentally one of the most effective way to improve error discovery is code reviewing which is promoted by Open Source and collaborative software development and discouraged by Closed Source. Imposing a chain of responsibility would have the only effect to advantage those producers with a legal team at their disposal and will have no real impact on the software development process at all.
The proposal excludes Open Source software only if it is non profit. This is a huge misunderstanding of how Open Source development works: Open Source means that the code is open to everyone to use and contribute (under specific licenses), it doesn’t concern the way it is eventually profited on. By imposing a chain of responsibility the law hurts any collaborative software production at its core.
Every modern software project is a composition of thousands of different independent modules with many different levels of interaction between them that constantly change over time and it is impossible to track them all (it’s a tree, not a list and the developer cannot act on levels lower than the one it is working on).
A collaborative software effort, like Open Source, accepts free contributions by everyone but only some individuals or organizations might monetize it. It is thus impossible to share the same responsibility about bugs, errors or malicious intents. That would in fact hamper collaborations at all for fear of being involved in legal issues. It is even impossible (or disproportionately expensive) to analyze every contribution to be able to (sort of) guarantee anything about that legally (i.e. the linux kernel has over a million contributions).
Bugs are often due to the unintended interactions between codes written by different people often on different modules and on a different timeline! To pinpoint the responsibility of an error to a single developer and being able to assess its maliciousness might be a serious legal and technical problem that would practically void the effect of the legislation.
An effect of this legislation would be that bugs will not be disclosed as freely as they are today for fear of being put under investigation: this is completely against the aim of fighting errors and improving security.
This legislation imposes a legal requirement that it’s easily affordable by big companies but might present important concerns for the smaller ones. For big companies fines are often budgeted alongside expenses and they have inside legal help too. The same is not true for lone developers or small software companies for which it’s a huge, possibly inadmissible, risk.
This proposal seems to follow the same intent as the software copyright law: to establish a de facto dominance on the market by a few big companies by rising the requirements to a level that is impossible to satisfy by any smaller actor. This is blatantly against the principle of concurrency and free market.
For any practical effort this proposal is just a sort of quality management process to software development (plus huge penalties) and the quality legislation should be enough to cope with that in the cases where security should be particularly cared of.
Note
This is a work in progress article on a matter that requires urgency so expect this text to change quite often. Feel free to let me know if you like it and if it can be improved in any way. Feel free to share as much as possible (my job and that of many others might depend on that)!
Databases need a unique identifier called primary key (PK) to address their data units (records). There are two types of primary keys: natural keys, which are extracted from the data (i.e. first and last names or the social number), and surrogate keys which are sequential serial numbers added externally (usually by the database itself) and called sequential ID.
The most common form of sequential ID is an 8 byte integer which is very compact and performant in the context of a single database but it’s not very helpful in a distributed environment where key uniqueness among all systems is paramount. It is analogous to the difference between a local serial number added to a book by a library and the ISBN code: the former is useful, practical and efficient within the local indexes and users but only the latter can be used effectively to exchange books with other libraries.
Solutions for a distributed system
We need an identifier that maintains its uniqueness in a context with many different interacting systems. There are three solutions available:
An hybrid natural/surrogate key model
A centralized identifier generator factory
Universal identifiers
Hybrid natural/surrogate key model
If the data unit has a fitting natural key that is assured stable, unique, easily indexable and compact it might be used as an external identifier while still using a surrogate key internally to keep the database performant. The natural key could be substituted by an UUID (see later) with similar results. Applying this method would cost an extra index to be paid in speed and space and the different databases involved are bound to be managed separately because of the non unique primary key. This option might still be viable for loosely interacting systems that are managed separately (i.e. in a wide area network vs cloud). Natural keys are commonly discouraged though because of the many concerns about stability, security and privacy (it might be a Personal Data).
Centralized identifier generator factory
A centralized identifier generator factory can be a solution where a strict control over identifiers is needed (i.e. authorizations). It requires specific measures to address performancebottleneck, security and availability.
Universal identifier
A big random number can be assigned as an identifier by every system independently. A commonly used one is the UUID: Universally Unique IDentifier.
Random UUIDs are guaranteed unique (no need for a centralized factory), secure (non guessable), compact (16 bytes long) and easy to use but they have one huge problem: their own randomness! They cannot be efficiently indexed by the B-Tree algorithm used by most databases leading to a significant drop in performances compared to sequential IDs.
On the other hand universal identifiers, avoiding key collisions over multiple systems, improve considerably the management of a distributed data layer allowing for a far easier data migration, merging, distribution and backup.
Sequential UUID
To address the B-Tree indexing problem there is a proposal for an extension to the UUID specification to include monotonically incremented, node and time dependent UUID versions (sequential UUID, v6, v7, v8) that improve indexing performances dramatically. Note that UUID version 1, incrementing its most significant bits, doesn’t get those improvements.
Unfortunately even sequential UUIDs, being double the size of a normal ID, suffers some performance loss for the intrinsic slowness of some common operations like sorting by ID and a noticeable increment in the overall database size.
One possible solution would be to use a sequential UUID as a secondary index and a normal sequential ID as a primary key. This is based on the premise that an object will be referred much more internally by means of inner relationships, indexes, views, caches etc than from external systems (internal cohesion). This is very similar to the hybrid natural key-surrogate model presented before, it only uses the UUID as an address to be searched externally but it completely misses any advantages it could bring to the data layer management.
Small and efficient: TSID
One interesting solution is TSID which is a monotonically incremented, node and time dependent universal identifier of only 8 bytes (against the 16 bytes of a standard UUID). Being sequential and of the same size of a normal ID it keeps all its benefits in terms of efficiency on the database. Being universal it’s unique among all involved systems with benefits on the distributed data layer that sequential ID doesn’t have.
TSID mimics and extends the notorious Snowflake ID which was created by Twitter and now used by many others including Instagram and Discord. It’s very configurable. Its 64 bit payload is divided between a node identifier ranging from 0 to 20 bits (0 to 1,048,576 maximum nodes), a millisecond accurate timestamp of 42 bits (will be running for the next 139 years) and a counter of 2 to 22 bits for infra millisecond generations (22 – node bits, able to allocate from 4 to 4,194,304 TSIDs for each millisecond). The more bits are allocated to the node, the less are available for the infra millisecond counter but the generator can allocate values from the next milliseconds mitigating this problem if a generation burst should happen. This is probably its only flaw: given the right condition (low assigned infra-millisecond counter bits) it’s fairly easy to guess the next sequence value. A way to prevent that is to export it encrypted (at the cost of losing its sortability).
Note that because it uses all of its 64 bits the resulting long numbers cannot be managed by Javascript and must be encoded into a string (TSID has an helper that returns a 13 characters string representation of itself). That’s because a Javascript number is a IEEE a double-precision 64-bit binary format IEEE 754 value and its integer part can only store 52 bit (see Javascript Number.MAX_SAFE_INTEGER), a bigger value will be rounded (with imaginable consequences). The upper limit of 252 is a huge number (more than 9⋅1015) and will not pose any practical limitations for long sequential IDs (that’s why you have probably never heard of this problem before).
Being in a sequence and because of its very limited bit count it’s vulnerable to next sequence value guessing attack. This means that given a TSID, under some conditions, the next values could be guessed quite easily. To avoid this, and if required, TSID can be encrypted before being sent externally at the JSON serialization phase (API boundaries).
TSID is a monotonically incremented, node and time dependent universal identifier of only 8 bytes (against the 16 bytes of a standard UUID). Universal means that, under specific conditions, it can produce unique values on different independent systems. Being the same size of a long and generated sequentially it keeps all the benefits of the classic long ID as a primary key on a database (and other storage systems as well).
TSID mimics the notorious Snowflake ID which was created by Twitter and now used by many others including Instagram and Discord. Its 64 bits payload is divided between a node identifier ranging from 0 to 20 bits (0 to 1,048,576 maximum nodes), a millisecond accurate timestamp of 42 bits (will be running for the next 139 years) and a counter of 2 to 22 bits (22 minus the number of bits used by nodes) for infra millisecond generations (able to allocate from 4 up to 4,194,304 TSIDs per millisecond). The more bits are allocated to the node-id, the less are available for the infra millisecond counter but the TSID generator can allocate values from the next milliseconds thus mitigating the problem if a generation burst should happen.
For a more detailed explanation of why TSID is useful in the context of a multi system environment see my other article “The primary key dilemma: ID vs UUID and some practical solutions” (yet to be published).
Factories
There are various factories available in the package and some might have unexpected results if misused which means having multiple non unique TSIDs generated.
We should consider two different environment:
within the same node (different threads on the same machine)
on different nodes
To check which is the best way to use the available factories take a look at the output of the tsid-factory-test application. It generates concurrenlty with each factory strategy and analyzes the results thus also simulating generation on different nodes. The following values were obtained running the TsidCollisionTestApp using 32 threads on a 4-cores, 8-threads CPU, your actual values might be different but should be consistent:
Description (Function<Integer,TsidFactory> in the 2nd line)
duplicates
op/ms
1
Create a new default TSID factory on each thread i -> TsidFactory.newInstance1024()
220610
27211.0
2
Share the same default TSID factory on each thread i -> sharedInstance1024
0
5202.0
3
Create a new TSID factory on each thread with the same node-id i -> TsidFactory.newInstance1024(0)
14568357
38278.0
4
Share the same node TSID factory on each thread with the same node-id i -> sharedInsstance1024Node0
0
4468.0
5
Use a different TSID factory on each thread with a different node-id i -> TsidFactory.newInstance1024(i)
0
18605.0
6
Use a new thread local random TSID factory on each thread i -> factoryCreator()
204625
36281.0
7
Share the same thread local random TSID factory on each thread i -> sharedFactory
0
4291.0
We can immediately see that using a different factory for each thread results in duplicated TSIDs and must be avoided (cases 1, 3, 6). For the same reason these cases are bad for multi node systems as well. The only case where using different factories works is when a different node-id is assigned to each one and that’s the case to use on different nodes (case 5).
Within the same node all threads should use the same shared factory. In case of a multi node system nodes should use TsidFactory.newInstance1024(nodeId)(or equivalent depending on the desired maximum nodes) while within the same node all threads should use the same shared factory with the same node-id.
Because the factory TSID generator method is synchronized (for good reasons) there is no need to use a ThreadLocalRandom custom generator (as hinted in the docs) that, running inside a synchronized block, will not improve the performances in any way (case 7).
NFS is an old protocol used to share filesystems among many unix systems.
Pros:
It works with almost all unix flavours and is also supported by other systems (there are Windows and Mac clients available)
It’s quite fast and well optimized.
It fully supports unix users, privileges and ACL.
At first it used UDP but the latest versions use a single TCP connection to a fixed port (IANA TCP/2049) which makes it very easy to tunnel through SSH considerably enhancing its poor security (see cons).
It’s been designed to support a lot of clients, which makes it ideal for supporting large networks (i.e. parallel computing environments such as MPI or labs configurations where many clients access the same disk managed by a server).
Cons:
It doesn’t have any type of reasonable security by itself (neither by access nor by resources). It filters allowed clients by source IP address only (to make it possible attaching ‘stupid’ clients). This makes attacks trivially simple given an access to the same network.
The ssh tunneling option (see later in the article) greatly enhance transport security and authenticationbut any root on any client might access and modify all files in the shared folder so detailed access control should be made at the shared folder level. The root_squash option manages if clients can actually operate as root on the shared folder.
Fom exports(5): Very often, it is not desirable that the root user on a client machine is also treated as root when accessing files on the NFS server. To this end, uid 0 is normally mapped to a different id: the so-called anonymous or nobody uid. This mode of operation (called root squashing) is the default, and can be turned off with no_root_squash.
SMB
SMB is a protocol from Microsoft that has been made available to unix via the SAMBA project.
Pros:
It works well with almost all versions of MS-Windows and other systems as well (Mac, android, *nix).
It supports various type of strong authentications and encryptions.
It’s reasonable fast and efficient.
It has limited support for unix clients.
Cons:
Unix extensions are still lacking full support (to a unix machine all files appears belonging to the same user and executable and there might be other limitations depending on the implementation). This prevents fully integration between unix systems: it’s really not a good way of sharing a filesystem but it might be ok for documents, images and office data.
Its access control model doesn’t adapt well to unix’s.
There might be problems with character encodings especially with older Windows systems.
Comparison
Scenario
I needed to build and run a bunch of java microservices on a separate machine while I use the main one for developing. This is because the full development environment plus browser, git tools and accessory programs fills almost all the available memory (16 Gb) leaving very little to run the system under development as well (db, Keycloak, microservices and Angular front-end plus docker related stuff and monitoring). The catch is obviously that the two systems should share the same folder containing code and configurations (the microservices are instructed to recompile themselves upon every code change).
I made several experiments with different configurations. This isn’t a very detailed test (the differences are so prominent that I didn’t feel the need to go deeper with details). It’s based on the most up to date software available on October 2022 on Ubuntu 22.04.
Test
These comparison times refer to a full compilation of a Java project with tests (involving starting a real postgresql instance in docker). This implies a lot of reading and writing of small textual files and the compilation of 130 java sources. The machines are connected to a 1 Gbps network by a switch and are:
Main: Notebook with Intel I7-2630QM 2.0 Ghz CPU and 16 Gb RAM running Kubuntu 22.04.
Secondary: PC with Intel I7-2600 3.4 Ghz CPU, 14 Gb RAM running Ubuntu Server 22.04 LTS in a VirtualBox 6.1 virtualized instance with 8 CPUs and 8 Gb of RAM under MS-Windows 10 Home host.
mount -t cifs -o username=fra,uid=1000,gid=1000,noperm,fsc,sfu,serverino //192.168.1.1/devel /media/devel
8
SMB3
mount -t smb3 -o username=fra,uid=1000,gid=1000,sfu,fsc //192.168.1.1/devel /media/devel
7.3~8.4
NFS over SSH
open the session with tunnel on server: ssh -R 3049:localhost:2049 server-name connect from client (in the SSH session): mount -t nfs -o port=3049 localhost:/media/devel /media/devel
Results
It’s clear that SSHFS is the slower option: it’s mostly an hack that runs on top of the SFTP protocol so it’s very lacking as a general purpose filesystem. It’s not even that bad if all you need is the occasional browsing and picking of files but it’s definitely not up to the task of being a real shared folder. It’s very easy to setup being readily available wherever an ssh server has been installed. Its sustained transfer speed is not bad at all but it suffers on browsing directories with many files.
CIFS is just the old protocol for SMB while SMB3 is its latest version. SMB3 is obviously much faster than CIFS. If it wasn’t for the lack of a proper unix privilege and user support it would be mostly ok (easy to configure being present by default on most systems).
NFS tunneled into an SSH connection is the real winner here! NFS is a good network filesystem but lacking any form of security and SSH just adds on that giving back the best possible solution. Considering the presence of the virtualization layer and the tunneling encryption being only 50% slower than the local access is quite impressive (there is CPU work involved here that I haven’t rightly accounted for but its a working rough estimation).
Installation of NFS + SSH tunnel
Please refer to this guide to install and configure NFS on SSH (which is considered already installed). It’s pretty easy and straightforward.
1. Install NFS on Server Install the required packages (Ubuntu 12.04):
mount -t nfs -o port=3049 localhost:/home/proudlygeek /mnt/nfs-share
Notes
The default docker installation on Ubuntu (which uses snap) has a bug (or a security feature, it depends) that makes it impossible to run docker (and consequently docker-compose) from a shared folder: docker should then be removed as a snap and installed directly from docker as a package.
In the latest versions the stand-alone docker-compose program has been moved to be a plugin of docker so to call the former docker-compose one should now use docker compose (note the space instead of the dash). Just define an alias: alias docker-compose='docker compose'to compensate for that.
The docker image for PostgreSQL is configured to have its database on an external volume on the local filesystem (which is actually the shared NFS folder). Because NFS is by default forbidding writing as root and postgresql needs to do that, the docker image creation fails. To make it work the NFS exports should have no_root_squash parameter set.
Disclaimer
This article is the result of a brief exploration I did to solve a specific problem at hand and I decided to write it down because I didn’t find any other place where these facts were explained plainly and simply (many seem not to care about the SMB problems with unix). So that’s it. I am not really an expert on the matter, if you have any further clarification or have discovered any error or inaccuracy feel free to contact me.
Recentemente Whatsapp ha aggiornato la sua privacy policy con un’informativa che aggiunge l’uso della geolocalizzazione e lo scambio di dati con aziende partecipate di Facebook (proprietaria di Whatsapp). L’obiettivo di Facebook è ovviamente quello di monetizzare una piattaforma per la quale ha sborsato ben 19 miliardi di dollari nel 2014 e che ha costi di servizio astronomici e intende farlo con le seguenti strategie:
Servizi dedicati alla clientela a pagamento business;
Comunicazioni commerciali con aziende tramite chat automatiche (bot) per acquisti e servizi di informazione e assistenza;
Geolocalizzazione per suggerire business nelle vicinanze;
Integrazione con il mondo Facebook e monitoraggio delle relazioni sociali
Facebook è essenzialmente una piattaforma pubblicitaria che promette ai propri clienti una platea mirata con un alto ritorno di investimento. Per fare questo Facebook profila attivamente i propri utenti e li caratterizza in base alle preferenze (likes), ai post, ai commenti, ai gruppi a cui si partecipa, alle persone con cui ci si intrattiene e a molte altre informazioni statistiche che rivelano moltissimo circa ciascuno di noi. Facebook nega che queste informazioni vengano cedute a terzi (è il loro asset d’altronde), ma nonostante ciò ci sono state operazioni controverse in passato come quella riguardante Cambridge Analytica a cui sono stati ceduti 87 milioni di profili (un abuso secondo Facebook ma la questione è molto controversa). Questi dati sono stati poi utilizzati per operazioni di influenza politica e fake news.
E’ questo il punto più delicato della questione: siamo abituati a considerare la profilazione commerciale relativamente innocua e forse perfino utile ma rischiamo di non percepire il rischio di vivere in un mondo creato su misura per noi con l’obiettivo di venderci merci, servizi e idee. Rischiamo di non capire che quello che vediamo sui social (come sui motori di ricerca) non è lo stesso mondo che vede qualsiasi altro e questo distorce inevitabilmente la nostra percezione della realtà. Viviamo in universi cognitivi isolati e manipolati dai professionisti della vendita. L’elevata polarizzazione ideologica di questi ultimi anni è solo uno dei drammatici effetti di questa distorsione.
La normativa europea GDPR intervenuta per proteggere i cittadini dell’Unione dalla raccolta e commercializzazione massiva di dati personali non vieta la profilazione, ma semplicemente obbliga chi la fa ad essere trasparente su chi profila, cosa profila, come lo fa e a chi finiscono i dati raccolti e ad un bilanciamento fra i diritti degli utenti e il legittimo interesse (economico) di chi la compie. E’ vietato che la sola profilazione possa essere usata per discriminare ma di fatto è un confine legale molto labile. Insomma la normativa pone degli utili paletti ma è ben lungi dall’arginare il fenomeno soprattutto nelle sue espressioni più sotterranee.
Torniamo dunque alle piattaforme di messaggistica. Intuitivamente siamo portati a ignorare il mezzo di una comunicazione e a concentrarci sul destinatario. Ma cosa succederebbe se il nostro prezioso messaggio fosse letto da altri a nostra insaputa? O che lo sia anche solo il fatto che corrispondiamo con qualcuno con una certa frequenza e magari specialmente la sera tardi (metadati). L’elenco dei nostri clienti potrebbe essere molto utile ad un nostro concorrente o il fatto che corrispondiamo con un attivista politico potrebbe renderci target per informazioni ideologizzate.
Il mondo della finanza si basa sulle statistiche da cui si estraggono informazioni circa il ritorno degli investimenti, in particolare le assicurazioni. Cosa succederebbe se la tua assicurazione sapesse che sei un appassionato di modding di automobili da corsa? Non è passato troppo tempo da quando i recruiter chiedevano di accedere alla pagina Facebook prima di assumere. Te lo concederebbe la tua banca un mutuo se sapesse che hai il vizio del gioco? Sono ipotesi irreali? Sono informazioni troppo ghiotte e la legge sta sempre un paio di passi indietro: aspettatevi che in qualche modo lo facciano. Non creiamo i presupposti affinché si realizzi questa distopia antidemocratica: è per questo che laprivacy è importante.
L’impatto della profilazione dipende ovviamente dalla sua scala e quando arriviamo alla grandezza di Facebook stiamo parlando del mondo intero. E’ il momento di cominciare a preoccuparci? Secondo me sì (già da un po’). E allora cosa fare? Rinunciamo e diventiamo eremiti tecnologici? Certamente no, ma abbiamo un’arma molto efficace: la differenziazione. Cominciamo a cambiare piattaforma di messaggi con qualcosa di analogo ma fuori dalla portata di un colosso della profilazione come Facebook. O quantomeno facciamolo per alcuni messaggi.
Offerte commerciali: Skype, Telegram e Viber
All’epoca dell’esplosione di Whatsapp la messagistica mobile non esisteva: c’erano gli SMS, e per il PC c’era Skype o Facetime (e prima ancora ICQ). Su Skype ogni utente era rappresentato da uno pseudonimo (username) scelto a piacere e per parlare con qualcun’altro si doveva conoscere il suo pseudonimo. Al passaggio sul mobile lo scambio degli pseudonimi cominciò a diventare una complicazione fastidiosa che fu abilmente scavalcata da Whatsapp con una manovra che all’epoca fu molto coraggiosa: leggere la sacra, privatissima, rubrica telefonica del cellulare. Una scelta strategica che poi ha comportato il legame indissolubile di ogni utente con quell’unico numero telefonico (e varie altre complicazioni). Grazie a questa trovata, e con grande stupore di Google che non aveva (ancora) osato tanto, Whatsapp si diffuse enormemente. Poi, di fronte ai primi problemi di privacy arrivò finalmente la cifratura E2E (da utente a utente), la (pessima) applicazione desktop, la voce e infine il video. Il modello di business iniziale era a pagamento (pochi euro all’anno) che fu presto abbandonato per puntare all’acquisizione operando in perdita fino all’acquisto nel 2014 da parte di Facebook (gran parte del costo fu per saldare i debiti). Non potendo accedere al contenuto delle conversazioni (protette, come si diceva, dalla cifratura E2E) Facebook ha deciso di profilare i metadati (ovvero qualsiasi cosa rimasta fuori) e la geolocalizzazione e aggregare questi dati al profilo Facebook. E siamo ai nostri giorni.
Skype è oggi di Microsoft ed essendo allegato ad ogni copia di MS Windows 10 è piuttosto diffuso nel mondo PC. Non ha invece molta penetrazione fra gli utenti mobili in parte per la sua impostazione seriosa e business. In questo settore vincono certamente le faccine, le animazioni e le spiritosaggini da cui Skype si è sempre tenuto distante. Di certo Microsoft opera con le medesime attività di profilazione di Facebook e considerando la portata della società c’è una sostanziale equivalenza delle problematiche.
Telegram è un prodotto russo il cui modello di business è la pubblicità nei canali pubblici (una sorta di chat di gruppo con moltissimi lettori passivi). Ha avuto un discreto successo commerciale grazie soprattutto agli stickers (c’è tanto da imparare da questa lezione) e un’interfaccia estremamente curata e amichevole. La sua strategia tecnologica si basa sulla centralizzazione dei dati sui propri server, e questo se da una parte ne permette l’uso senza particolari difficoltà da diversi dispositivi contemporaneamente, dall’altra concede al gestore la possibilità di leggere virtualmente qualsiasi messaggio (ha la cifratura E2E solo come opzione, con molte restrizioni e non per le chat di gruppo). Il fatto che non li leggano rimane un grande atto di fiducia a cui si danno molta pena di aspirare. A seguito di una legge russa circa l’obbligo di concedere l’accesso da parte del governo alle comunicazioni private la società ha pensato bene di trasferirsi (l’alternativa era ovviamente chiudere per fuga in massa degli utenti) in un posto di rinomata garanzia democratica: Dubai (UAE)! La loro applicazione è Open Source (che è un bene ma il livello del codice sembrerebbe piuttosto mediocre) e si basa su algoritmi aperti (ma mai oggetto di review) ma questo non cambia il fatto fondamentale che tecnicamente abbiano accesso al contenuto dei messaggi.
Viber è un prodotto commerciale che viene monetizzato tramite l’acquisto di estensioni (per esempio stickers), giochi, chiamate ad apparecchi telefonici e la pubblicità sia nell’applicazione che nelle chat. Pur essendo un prodotto maturo, ben provvisto e con una cifratura efficace il suo modello di business non è esente da critiche sulla privacy (d’altronde vendono pubblicità).
Signal
Vieniamo finalmente a Signal: è di una società no profit finanziata da una fondazione e dalle donazioni che riceve, il suo codice è Open Source (quindi leggibile, verificabile e compilabile da chiunque) ed è stato sottoposto a verifiche di sicurezza da parte di esperti del settore di rinomata fama e affidabilità, usa sempre la cifratura E2E di dati e metadati, ha tutte le caratteristiche di tutti gli altri, perfino le faccine e gli stickers, vari client per sistemi operativi, voce, video, trasferimento dati, è consigliata dai maggiori esperti di sicurezza come Bruce Shneier ed Edward Snowden (scusate se è poco) e in ultimo… funziona benissimo! Suona troppo bello? Beh, di certo non vedremo servizi business su questa piattaforma ne modi di scambiarsi valuta (forse Bitcoin, non saprei) ma è certamente una grande opzione per la messaggistica privata che intenda restare tale. Unica piccola concessione alla facilità d’uso è l’uso dei numeri telefonici per identificare i propri utenti: da una parte semplifica molto il problema della ricerca e identificazione dell’interlocutore, dall’altra intacca la privacy. Detto questo un cellulare con un numero di telefono usa e getta è sicuramente nella valigetta di qualsiasi 007 che si rispetti…
Per chi ha voglia di approfondire suggerisco quest’ottimo sito.
I needed to configure a small laptop for secure out of office working and I decided to go full on security and features and to learn something along the way. That is, using the full disk encryption of the Lubuntu installer and try the (in)famous btrfs filesystem.
The laptop is a Sony VAIO VGN-TT1LN with an Intel U9300 CPU, 4 Gb RAM and 120 Gb HDD found fully working in a rubbish dump (that’s crazy right?).
I just used the default full disk encryption option offered by the latest 20.04.1 Lubuntu installation but instead of allowing one single big partition as the default installation suggested I preferred a two partition installation with a partition reserved for swap. I thought it could help to cope with hibernation but it might be just not required at all. Anyway I manually partitioned the HDD into two main partitions:
a 5 Gb LUKS encrypted linuxswap;
a LUKS encrypted and BTRFS formatted (flagged as root).
I chose the auto login option because LUKS unlocking password would already protect the access to the system anyway. The password is still required in case of restart or logout to other accounts.
Once the installation process finished (around 20 minutes) I found a perfectly workable system secured with aes-xts-plain64 with a key size of 512 bit. Unfortunately the installation didn’t allowed me to specify a preferred encryption algorithm, I would have probably preferred serpent that appears (cryptsetup benchmark) to be about 20% faster than aes on my system (there is not aes instruction available to the Intel U9300 CPU). Anyway aes performances are around 77 Mb/s for both reads and writes which are good enough so I didn’t bother to try to change that.
Hibernate
Unfortunately a little bug in the installation prevented the system to hibernate properly. There was the wrong partition UUID specified as RESUME (probably it was set for the one partititon only default configuration). I solved that following the instructions I found here of course substituting /dev/mapper/ubuntu… with my swap device:
Btrfs is a wonderful new filesystem that offers incredibly useful features (at the cost of being not quite super stable yet). Unfortunately it doesn’t support native encryption (ZFS does) and so one must use dm-crypt for that (which is an acceptable compromise). In respect to ZFS, which is a mature COW filesystem created by Sun and now supported by the open source community with a CDDL (GNU incompatible) license, it offers much more flexibility in managing clusters of devices without even requiring to be remounted for most of its structural operations. It is really a life changer for sysadmins and advanced users.
The most powerful new concept introduced is the snapshot. It allows to fork the filesystem and proceed to change each branches independently. This is done atomically using the same data without the additional duplication cost and only recording for changed data. That’s great for safe experimentations and backups.
A snapshot is a special case of a subvolume and can only be taken from another subvolume. A subvolume is a sort of special directory that can be mountable as Virtual File Systems (VFS). Subvolumes can be created wherever a directory can, they can be nested, deleted, renamed and of course extracted as a snapshot.
Another interesting features of btrfs is that transparent compression and copy on write (cow) can be enabled on a directory basis by using respectively:
These attributes are inherited by the hierarchy but apply on new created files only (or you can run the command btrfs filesystem defrag). Of course the reverse apply if using -c or -C instead.
Layout
There are many layouts proposed to take full advantage of BTRFS snapshot feature, Ubuntu, as many other distros, proposes a simple scheme with @ for the root and @home for user’s homes. This is good but not optimal.
I propose the following schema:
@ for the root filesystem /
@home for /home
@local for /usr/local where I keep my JAVA manually managed environment and other manually installed stuff (python, node…)
@opt for /opt where some programs like to install themselves
@root for /root (useful to keep administrative stuff that should not be rollback-ed)
@var for /var to keep things that follow a different life cycle than / such as logs, databases etc or don’t need to be saved at all (spool, caches).
The following shell script helps to prepare the aforementioned structure:
#!/bin/bash
# gets the default ubuntu btrfs subvolumes (/@ for root and /@home for /home)
# and transforms them into a more sensible one
#### must execute as root!
if [[ "$1" == "" || ! -e $1 ]]; then
echo "device not found"
echo "usage $0 device-name user-name"
echo "es: $0 /dev/sda1 fra"
exit 1
fi
if [[ "$2" == "" || ! -d "/home/$2" ]]; then
echo "user home not found"
echo "usage $0 device-name user-name"
echo "es: $0 /dev/sda1 fra"
exit 1
fi
echo "parameters ok:"
echo "dev: $1"
echo "usr: $2"
echo ""
echo "mounting btrfs root into /mnt"
umount /mnt
mount -t btrfs $1 /mnt
echo ""
echo "doing a brtfs snapshot of root filesystem"
SNAP=/mnt/@.snapshot
btrfs subvolume snapshot /mnt/@ $SNAP
echo ""
echo "make a copy of fstab"
cp $SNAP/etc/fstab $SNAP/etc/fstab.$(date -Iseconds)
echo ""
echo "set /tmp on tmpfs"
echo "tmpfs /tmp tmpfs noatime,size=1G 0 0" >> $SNAP/etc/fstab
echo ""
echo "creating subvolumes"
for i in /opt /root /var /usr/local ; do
NAME=`basename $i`
SUBVOL=@$NAME
btrfs subvolume create /mnt/$SUBVOL
echo " adding to fstab"
echo "$1 $i btrfs subvol=$SUBVOL,defaults,noatime,space_cache,autodefrag,compress=lzo 0 2" >> $SNAP/etc/fstab
echo " moving files"
rsync -aHAX $SNAP/$i/ /mnt/$SUBVOL/
echo " remove original"
rm -Rf $SNAP/$i/*
done
echo ""
echo "setting nodatacow"
USERHOME=/mnt/@home/$2
for i in $USERHOME/.cache $USERHOME/.mozilla $USERHOME/.thunderbird $USERHOME/snap $EXTRAVAR ; do
echo " $i"
mkdir -p $i
chattr +C $i
done
echo ""
echo "done."
echo "Remember to add compress=lzo to /etc/fstab mount options."
I encountered this error lately when trying to upgrade my OpenProject docker compose from a perfectly working version 9 to version 10 (latest). Here is my solution based on the answer to this StackOverflow question. This is a simplified version of the docker-compose.yml I am using (I use letsencrypt):
When launched with the command docker-compose up the following error will appear in the log:
openproject | Creating scope :order_by_name. Overwriting existing method Sprint.order_by_name.
openproject | App 403 output: hook registered
openproject | App 403 output: Creating scope :order_by_name. Overwriting existing method Sprint.order_by_name.
openproject | [ E 2020-07-04 12:54:42.8702 364/Tj age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /app: A timeout occurred while starting a preloader process.
openproject | Error ID: 673f1464
openproject | Error details saved to: /tmp/passenger-error-RB2YJS.html
The problem is that the default timeout of 90 seconds is not enough for the process to start and the spawn operation is aborted. It suffices to change it to 300 in the /app/docker/web file inside the image. So copy the file from the image to your docker-compose.yml folder:
docker cp openproject:/app/docker/web web
And add in the last line of the passenger command --start-timeout 300 as for the configuration reference:
#!/bin/bash -e
BIND="${BIND:=0.0.0.0}"
PORT="${PORT:=8080}"
RAILS_ENV="${RAILS_ENV:="development"}"
MIGRATE="${MIGRATE:="false"}"
MIN_INSTANCES="${PASSENGER_MIN_INSTANCES:=1}"
MAX_INSTANCES="${PASSENGER_MAX_INSTANCES:=3}"
SPAWN_METHOD="${PASSENGER_SPAWN_METHOD:=smart}"
USE_PUMA="${USE_PUMA:="false"}"
if [ "$RAILS_ENV" = "production" ] && [ "$USE_PUMA" = "true" ]; then
# @TODO Add apache config to serve the assets. Until then we have Puma serve
# them which is slower. This is ok if an asset host is used (SaaS).
# But for self-hosted installations we may want to fix that.
# Passenger does include a config to have nginx serve the assets.
export OPENPROJECT_ENABLE__INTERNAL__ASSETS__SERVER=true
fi
if [ "$MIGRATE" = "true" ]; then
echo "Migrating database..."
bundle exec rake db:migrate
fi
if [ "$USE_PUMA" = "true" ]; then
# see `config/puma.rb` for configuration
bundle exec rails server puma -b $BIND -p $PORT
else
exec bundle exec passenger start \
-p $PORT \
-a "${BIND}" \
--min-instances "$MIN_INSTANCES" \
--max-pool-size "$MAX_INSTANCES" \
--spawn-method "$SPAWN_METHOD" \
--max-preloader-idle-time 0 \
--start-timeout 300
fi
Now update the docker-compose.yml so that the modified web file will be automatically used with new openproject images as well:
Fillumina's Blog 