[ADC-Ext 1.0.8] RDEX - Redirects Extended

This is Crise’s proposal for RDEX.

=== RDEX - Redirects Extended

The goal of this extension is to enhance the redirect capabilities provided by BASE by adding support for two new redirect operations and a way for hubs and clients to communicate accepted redirect protocols. It also attempts to make the concept of redirects more unambiguous and less implementation defined. The two new types of redirects added are a multiple choice redirect and so-called permanent redirect as outlined below. These new redirect types are not necessarily mutually exclusive.

==== Redirect Security

Hubs should always aim to include as much information pertaining to a redirect as possible using other fields defined in BASE. Clients may, for example, rely on the availability of such information to check the origin of a redirect before making irreversible decisions, such as accepting a redirect as permanent as defined below.

Clients should not attempt to connect to an unreachable or otherwise invalid destination resources indefinitely or in otherwise abusive manner when responding to redirects. Whether clients respond to redirects using default user information or the same information as used on the hub that initiated the redirect is implementation defined, however, in the case that the same information is used careful consideration should be taken when it comes to automatically sending sensitive information such as passwords to other hubs especially if it provides less security to the user than the original hub.

Both clients and hubs should also take note that the TL field, as defined by BASE, should only impact the current hub and its connection (if re-established) thus it should not be relied on to trigger a delayed response to a redirect.

==== Accepted Redirect Protocols

Support for this extension is indicated by the presence of an RP field in the INF, which defines the accepted redirect protocols for the hub or a client as a bitmask using any combination of following values. Additional compatible values may be defined by other extensions.

|Protocol |URI Syntax |Flag
|User action (exclusive) |any supported, as defined below |0
|ADC (BASE) |adc://adress:port |1
|<<anchor-ADCS,ADCS Extension>>	|adcs://address:port |2
|http://nmdc.sf.net[NeoModus Direct Connect] |dchub://address[:port] |4

Hubs should treat the absence of the RP field in clients INF as the equivalent of RP with the value of one, thus implying support for BASE. If the ADCS feature is included in the clients feature list this may also be amended to include ADCS. If a client does not define an RP field the hub may choose to omit the RDEX syntax from redirects, although this extension retains full backwards compatibility with BASE. If a hub defines an RP field for itself it should prevent its users from redirecting others to protocols it did not define, including redirects to destinations without an explicit protocol handler. Hubs should always aim to use fully qualified resource identifiers for all redirects, regardless of whether an RP field is defined or not, to avoid ambiguous destinations due to implementation defined default handling.

Clients should refrain from removing support of a previously signalled redirect protocol from their RP field. It is up to the hub to decide a corresponding course of action should the client do so. Additionally a special case is defined for clients as an RP field with the value of zero to indicate that the client may not react to redirects as defined by BASE or this extension at all or may do so only as a result of user action. As such clients defining a non-zero RP field should follow redirects to accepted protocols automatically and conversely any attempts to redirect clients that define RP as zero should result in normal disconnection as per BASE.

==== Multiple Choice Redirect

Multiple choice redirect is defined as a redirect where instead of a single destination resource a set of resource identifiers is provided. The methodology that is used to select the final destination resource for a multiple choice redirect is implementation defined. However, only one final destination resource may be chosen. For example a particular implementation may prefer one protocol over another, encrypted connection over an unencrypted one or even simply choose one at random.

This type of redirect is signalled by the presence of an additional RX field in the QUI as defined for redirects by BASE. The value of the RX field is a comma separated list of resource identifiers to use in the redirect. The RD field should be defined for maximum compatibility and clients should add its value to the list of identifiers defined in RX when selecting the final destination resource.

==== Permanent Redirect

Permanent redirect is defined as a redirect with reference updating. In the case of a permanent redirect, if and only if a successful connection is made to the destination resource, the connecting party will then update all relevant references of the original resource to the destination resource. Examples of such references may be hublist records for pingers or favorite entries for clients.

This type of redirect is signalled by the presence of an additional PT field in the QUI, as defined for redirects by BASE, with the value of one. Upon receiving such redirect clients will proceed to update references as outlined above provided the conditions are met.

==== Examples

Multiple Choice Redirect (BASE compliant)
|Example |Description
|IQUI BBBB IDAAAA RDadc://example.com:1001 RXadcs://example.com:1002,adc://example.org:1020 |Here user BBBB is presented with a choice between three destinations Two on example.com and one on example.org. Old and non-supporting clients will be redirected to adc://example.com:1001. User AAAA is the originator for this redirect.

Multiple Choice Redirect (RDEX only, not compatible with old clients thus not recommended)
|Example |Description
|IQUI BBBB IDAAAA RXadc://example.com:1001,adcs://example.com:1002,adc://example.org:1020 |For clients supporting RDEX this is the same as sending the first example. Old and non-supporting clients will be disconnected and they might reconnect to the original hub after an implementation defined time.

Permanent Redirect
|Example |Description
|IQUI BBBB IDAAAA MSWe're\smoving! RDadc://example.com:1001 PT1 |Here user BBBB will be redirected to adc://example.com:1001 with the message of "We're moving!" and AAAA as the originator. BBBB should (see section Redirect Security) connect directly to adc://example.com:1001 in the future.

Permanent Multiple Choice Redirect
|Example |Description
|IQUI BBBB IDAAAA MSWe\shave\sADCS\snow! RDadc://example.com:1001 RXadcs://example.com:1002 PT1 TL0 |Here user BBBB is redirected permanently with a choice between ADC and ADCS URI, if and only if either of the URI's matches the URI of the current hub connection and the matching URI is chosen BBBB will reconnect immediately. Sending a redirect like this to all connected clients (once) can be used to attempt to migrate them to ADCS instance of a hub.

For historical reference, below are the different iterations of this on pastebin.

http://pastebin.com/uHFaF5B9 (current, asciidoc, same as in first post)

Why can’t the hub just send all available addresses when redirecting and let the client choose? Why is this information relevant to other users?

What’s the purpose of restricting the protocols? Why is an RP field needed if hub is the one performing the validation?

I’m not sure about this flag as it may cause public hubs to lose some their users permanently because of hubsoft/client vulnerabilities or bad permissions. Even if such flag existed, all clients (or outdated versions) wouldn’t be able to handle it. How is the hub excepted to handle those users?

There are still many hubs using older versions of FlexHub that allow anyone to send basically anything to the other clients (even with a false identity)… I’m not sure if there other hubsofts with similar vulnerabilities. If such flag is added, it would be more secure if clients accepted permanent redirections only before the NORMAL state.

If you read the text carefully, regarding permanent redirects and whether they are actually “trusted” is up to clients. In general, security in the end is the responsibility of the implementer anyways, the text already gives ample examples as to how implementations can deal with different aspects of it. However, as per discussion with Pretorian SHOULD and SHOULD NOT are preferred to MUST and MUST NOT which is why the final call is left to each implementation. Currently the wording of the text is such that while hubs can omit RP clients can also decline all or parts of this extensions functionality without it if they so choose.

Bottom line with security: if either side implementing this is dumb then that is not on anyone but them.

Currently my implementation, for example, ignores the permanent flag when it is sent in post-NORMAL state without an ID field referencing a user with CT flags for hub or owner. As for buggy flexhubs, it is unfortunate that they exist but simply the fact that they exist should not be an extension blocker. If that was the case then eventually nothing could never be added to the protocol. Re old clients and permanent redirects, hubs don’t need to handle that case, the command regresses nicely into BASE (ie. unknown flag is ignored). Hub can keep issuing redirects to old clients infinitely if it wants to but in practice that is something that is intended to eventually be deprecated by the existence of this extension.

RP is not relevant to other users, but it is relevant to the hub (a hub could filter out RP from INFs to other users, but that would be against BASE). And hub can send all available addresses as well, RP exists partly for the special value 0, and in the spirit of sharing information so that clients could cross-check hubs validation (I made the comparison to HTTP accepted headers in the hub). Also because QUI is fatal, sending it should be avoided if there is uncertainty (and to avoid such uncertainty more information is provided).

[23:54:54] i’d prefer no RP at all - then when a redirect occurs, the hub sends all its addresses and lets the client choose.
[23:55:10] that is intended
[23:55:35] but without RP the hub has no information on what the client will do
[23:56:49] and it should, again see RP0, hub networks in particular hate sending users to limbo, and private hubs prefer to have info on it (because some want that redirects are not followed)
[23:58:46] RP could be made into a boolean field to that effect, but why not give the hubs the option to micro manage if they want to
[23:59:14] the footprint won’t get any smaller anyways
[00:00:37] that’s convincing. > :slight_smile:

[16:48:00] The hub should send a QUI and then disconnect, nothing else should happen.
[16:48:49] There’s no “no, please don’t do it”

Well here are some comments out of my mind:

Whether clients respond to redirects using default user information or the same information as used on the hub that initiated the redirect is implementation defined, however, in the case that the same information is used careful consideration should be taken when it comes to automatically sending sensitive information such as passwords to other hubs especially if it provides less security to the user than the original hub.

We could send a named parameter (for example RA) for Reuse Authentication to tell the client whether he is supposed to reuse or not the Authentication information, as follows:
RA0: The client MUST not reuse authentication information.
RA1: The client SHOULD reuse authentication information.
No RA: The client can choose what to do, in general it would be a good idea to prompt the user.

Also, for non permanent redirections it could be useful to allow a TO parameter to be sent with the INF when connecting to the other hub (which can then be used by the other hub to validate the User’s identity without knowing his password). The way to do that though is outside of the scope of the standard (I can provide a proposal for a cryptographically sound ticketing generation system later though).

There is also something that called my attention and is that this feature is not negotiated at the initial handshake, negotiating the feature would allow old hubs to stay safe as the clients MUST discard the extended redirects if the hub hasn’t negotiated support for them.

I didn’t only mean old versions of FlexHub there… There are lots of different ADC hubsofts available and I’m not sure that how well the security of each hubsoft has been tested. People may develop new hubsofts that may experience similar issues. Even the existing hubsofts are modified and those changes could break the validation again. Clients have had issues with validating the incoming commands as well. Such validations are easier to miss in ADC when comparing to NMDC.

Redirection exploits have also existed in NMDC hubsofts (I can think of Ynhub) and it’s wrong to say that such exploits will never happen again in future. If such exploit is found by a wrong person, the hubs would be empty before the hubsoft developer even knows about the issue. Many public hubs wouldn’t survive if they lose all their favorite users. Allowing the hub owner to manually redirect some of the users permanently to a new address doesn’t feel that important, given the risks caused by accepting permanent redirections in NORMAL state.

I can also see various other issues related to this flag. It has been fairly common that the hub owner isn’t the only person who is able to change the hub config. Hub owners may give the access to people who know the hubsoft well or simply because they want give higher status to someone. Back in the days when I was actively running public hubs/a network, I used to have admin/owner access in most hubs where I stayed in. Generally I don’t think that it’s problematic at all because no admin could make permanent damage to the hub, and messing up the hub config just for fun doesn’t make much sense. Most hub owners won’t understand the change when such users could suddenly steal their users permanently. The current situation is somewhat clear when the hub owner must understand that all his users may be gone forever if he grants access to the owner account of the domain to someone else (and I don’t think that it happens often).

Even restricting the config access via the hub isn’t the only issue here. The hub owner may give his hub for someone with a fast connection/server for hosting. There are people who rent a common server that is used to host multiple hubs. I had a fairly unknown person hosting my hub for years after it was founded because I couldn’t provide an environment that was good enough for allowing the hub to grow. Later on, I could also gave a hub with 10k users for someone else to host temporarily when there were host issues (which wasn’t uncommon with that many users…). A hub may also redirect all its users elsewhere in case of temporary downtime.

When I ran the network, I also had people who were able to access the hub servers so they could help with setting up new hubs and performing updates. Many networks also used to have a common domain that was owned by the network owner. Permanent redirect would have made all those actions impossible because of the risk of losing all your users that you’ve been collecting for years.

I agree that in some cases it would be nice to get rid of old addresses when hubs are merged but the downsides/restrictions caused by this flag just seem much worse. Why couldn’t the extension just allow changing the protocol and the port?

That is how the concept of permanent redirects started out as, however, that original draft was basically ripped to shreds and I gave up on a it, abandoning the much narrower focus, and we end up with RDEX.

While not all the initial concepts problems affect RDEX as it is defined here, it still had more than enough of its own not directly related to execution. Pretorian is probably more than happy to give you the cliff notes on that discussion :stuck_out_tongue:

As for hubowners, I hate to be a bit rude, but just as I wouldn’t give my house key to a complete stranger same common sense should apply with high access accounts on hubs or any servers in general (that anyone cares about). Giving such powerful accounts to so many people implies that the hubs owner does not really care about the security of his hub. If he get screwed because of a poor decision he made then why should I as an extension developer so much as spare him a thought.

ADC is not intended to be a static protocol, change is expected encouraged even, so while I may consider faulty implementations when implementing (but not so much when creating) an extension bad user judgement is certainly not my responsibility in the slightest. Neither are hubs not updating their server software for example. Generally anything that the hub owner can do something about at all is not something I should be concerned things like access levels and which server software he is using falls to that category. Also because the protocol defines things like CT for INF and ID for QUI, it is perfectly reasonable that both of those are fields with some degree of trust if the owner or the server software knowingly forgoes or otherwise compromises tools defined to us by BASE there is not much we can or should do about it.

Currently ADC is not in a state where a main stream or reference implementations exists (particularly when it comes to hub software) either, so you can’t expect people to be aware of all the potential holes in old server software and clients or be aware of when someone is following BASE just little too literally.

Regarding redirection exploits, depending who you ask the ability to redirect itself is an exploit, RDEX does more to try and address this than it does add to it by a long shot (as opposed to base where redirect is no more than some unambiguous words).

Like I said above if we get too stuck on what clients could potentially do wrong with this if X, Y or Z, no progress can ever be made (because no matter how much we spend time dwelling on all the what ifs and maybes, when it comes to “bad” clients or hubs there is always another scenario that we didn’t prepare for it is just a matter of time). Protocol should be allowed to be developed with the assumption that baseline for clients and hubs is that they behave as expected and aren’t inherently bad or flawed. I have yet to see a suggestion as to how to fix the PT flag, that has not already been discussed, not remove it from your posts.

Hi guys, this is mainly the highlights of yesterday discussion with Crise, hopefully it can solve some of the problems mentioned:

  • Regarding the RP parameter:
  • To prevent certain attacks the lack of an RP parameter on the Hub INF should be considered the same as RP1 (which means support only base style redirects).
  • This still needs to be discussed with Pretorian though
  • RP0 is intended to mean no redirects allowed at all in this connection.
  • For added security the client must ignore any INF update from the hub changing the RP parameter (the hub can always do a drop or redirect if it REALLY is changed by the hub) when in NORMAL state.
  • Regarding the use of a FOURCC
  • Mainly discarded as the INF exchange could attain the same effect.
  • Regarding RA Flag (reuse authentication):
  • This flag makes no sense from a secuirty PoW as it will mainly provide inconvenciences and passwords can’t be deleted. Instead a recomendation should be given on how to prevent password leakage when implementing redirects with authentication token reusage.
  • Regarding TO (token to pass to the redirect destination):
  • The flag name will be changed to AT (authentication token) and will only be used on the QUI.
  • In order to ease simplicity this will not be mandatory instead will be signaled with RP flag 8.
  • When connecting to the destination hub, the hub will add the at= parameter to the RF URL followed by the token.
  • The token is a base32 string whose preceded by at least a base32 character used as identifier. Characters higher than 9 may acquire different meanings on the future.
  • Hubs can use tokens starting by the digit 0 (zero) in the way they want, others are reserved for future standards involving ADC hub to hub communication.
  • Token can be used to contain the following:
  • Authentication signatures by the source hub (so the destination hub can accept the user even if it is not in their user database without asking for the password).
  • Authenticated and encrypted user data for insertion in the new hub database
  • Any other data deemed interesting to transfer between the hubs
  • When combined with multiple redirect, AT can contain either one field with the data passed on all redirects or exactly as many comma separated fields as possible destinations. The first case means that the AT is the same for all the hubs, the second means that tokens from left to right are assigned first to the RD hub and then to the provided RX hubs.
    *Regarding redirect preferences stated by the hub
  • A new flag: LR (left to right ordering redirects) is defined, when defined the client shall ignore the RD field and try to connect to the hubs in the RX field (which will contain the hub in the RD field if necessary) from left to right until a succesful connection is made. The client can drop (or leave for latter) any protocols not supported (or not preferred). When not defined the client is free to choose the hubs from the list in the way he wants, including the one in the RD field.
  • One last thing was regarding the RP flags, in general representing them in hex simplifies parsing a lot as you can parse from right to left one digit at a time independently of the size of the flags record as opposed to having to convert the base 10 number into a binary large enough and then parsing it (for the record when using floats a la lua the maximum number of flags is 25).

Just to clarify regarding AT, the hub will not add anything to RF, because it can’t, but the client would of course be expected to do so.

LR flag was initially OR with the possible values of A, N or D (for ascending, neutral and descending respectively), however, while that is marginally more flexible it also presents with a problem of how strings are sorted (ie. natural sorting vs regular sort algorithms) and also optionally what is sorted (ie. the string address or the IP it is pointing to, which could be sorted as a number).

Regarding immutable INF fields, while this was discussed before extensively, simply requiring an RP field from hubs will not add any security in itself if it can be updated in NORMAL state. The issue regarding clients and their RP field is handled in the text as it is (ie. that it is hub decision how to handle updates), same could apply to hubs I suppose. Please consult the specific logs from yesterday in regards to this (and the original discussion re mutability if necessary).

As for the hex vs decimal argument, it is valid concern (and like I said on the hub a bitmask and its flags should be hex, because that is the common representation for a byte). However, it is important to make the connection to rest of ADC in regards to this. ADC uses bitmasks, f.ex. the CT field, and even there it is decimal across the board. I would also place criticism on that because the bitwise operations are not really the same as regular arithmetic addition (which is implied by the text in ADC, and the importance of the number sequence for the flags is thus left implied instead of explicit). A bitmask is a well defined concept and is not the same as simple addition of multiple numbers (but a subset of it, because the bits of the flags can’t overlap).

Also as an addendum to AT, in my opinion it should as the name implies only consist a token that may reference any manner of authentication related data. An individual tokens length should also be defined otherwise we will hit some implementation defined limitations regarding length of INF fields. Also naturally as with PID the token should not be broadcast to other users by the hub (it is one time use anyways, so it would have expired regardless) if because of this an independent INF field is preferred then that would also be fine.

The problem with defining the token as open data rather than a token with a specific signature is that extensions should not attempt to transmit data that is better to be transferred using say the file transfer protocols of ADC or some other hub to hub specific application of the ADC protocol. The bottom line, the token should be reference to data and not the data itself (in most cases it would be a one time use reference to some authentication profile defined elsewhere, but it should not contain said profile itself).

I won’t address the whole “oh my god, we might lose users”. We can’t forever prevent stupid hubowners from being stupid…

  1. Protocol/port vs full URI: The latter is always better because it is actually complete. It does not rely on in-text replacements and juggling around with RX addresses. If you want to add protection in the client, it is possible to do so with checking whether the address is correct or not for instance. Also, I generally would pull up a “This site wants to change stored information, are you sure you want to allow that?” or some scheme.

  2. A FCC is used to a) signal intent with functionality and b) filter out content to users based on that FCC. b) allows e.g. F-type messages which are highly useful (especially for the hub). In this case, the hub can, yes, filter clients in two categories whether they support or does not support RDEX. I don’t have an opinion here; if you want a FCC, then it’s not really a problem.

  3. Lack of RP from the hub for clients means… Just exactly what is does for now. It is not really the clients that should do the protection here… It’ll otherwise mean that it’d be impossible to move clients from e.g. a non-secure environment to a secure one…

  4. RP0 from hub, well, given that there is no redirect command (from a client), I don’t really see that as an issue… Just don’t allow +redirect or whatever?

  5. Sorry, I don’t understand the point of the token. What extra data are you giving that is useful? All of the use cases can be achieved without it, no?

  6. The LR flag means that clients cannot choose the sequence themselves, which is, I think, one of the better points with RX. E.g., the client may prefer a secure environment over a less secure one; the hub owner might not realize that their order is highly important. Such petty details is easier to let the clients decide. Instead, I think a suggestion in the spec that would make sense would be to have something akin to “Implementations may choose the order, but a left-to-right order is preferred.”

  7. Hex vs base 10? … ADC uses integers for everything else, and so this extension should also do so… If clients want to convert to hex in their implementation to simplify internal parsing or whatever, then that’s up to them…

This is just a post to make the connection between klondikes proposed AT change to RDEX and the linked extension draft more obvious because all the discussion on it was on the hub. I made an editorial judgement call to remove some of the useless banter from the quote, so for a full undiluted experience please ask around in the hub :wink:.

[2014-01-03 01:39] here is a crazy idea: add a QS param as part of RDEX whatever is in that (in standard query string syntax) should be forwarded to the destination as part of RF
[2014-01-03 01:39] As I said, I’d prefer if AT would be outside of REGX but I’m fine after our discussion to have it in REGX if we can find a good phrasing.
[2014-01-03 01:39] That’s fine, I guess, too.
[2014-01-03 01:39] leave all this tickets and tokens for another extension
[2014-01-03 01:40] the it won’t just be limited to “AT”
[2014-01-03 01:40] It would be good to have a better summary of our discussions at the forum, so people wouldn’t re-iterate the discussion.
[2014-01-03 01:40] however, the potential ramifactions re this and keyp should be considered… since keyp is information tied to the original source hub but this information would be transient
[2014-01-03 01:42] and it also means that RF will no longer match the URI client used to connect to the previous hub as well
[2014-01-03 01:43] which muddies the definition of RF in BASE
[2014-01-03 01:43] Agreed
[2014-01-03 01:45] I think that the idea of allowing hubs to embed extra information in redirects (without every single instance needing their own extension) has some merits though
[2014-01-03 01:45] We can always go back to the origin and pass the QS on the INF
[2014-01-03 01:46] klondike: true enough
[2014-01-03 01:47] but if making this more generic expands its scope while still allowing “AT” to be fully defined elsewhere and simply using this as its mechanism then I think that is worth considering
[2014-01-03 01:47] and this way a client doesn’t have to be aware of AT specifically just RDEX
[2014-01-03 01:47] For me that’s a solution that works
[2014-01-03 01:48] I’m OK with that.
[2014-01-03 01:48] So QS parameter which is passed on INF or QS passed on RF?
[2014-01-03 01:48] and we can avoid including the behemoth that is AT is shaping up to be from being part of RDEX
[2014-01-03 01:48] I would go with QS as an independant INF field
[2014-01-03 01:49] see conserns re RF’s definition in BASE

[2014-01-03 02:03] actually, we are complicating this unnecessarily,…
[2014-01-03 02:03] Maybe
[2014-01-03 02:03] if QS is a query string why is it not just appended to the url in RD and/or RX
[2014-01-03 02:04] then just have QS in INF contain that for the hub
[2014-01-03 02:04] yeah that makes sense
[2014-01-03 02:04] since client does not currently otherwise provide that to hub
[2014-01-03 02:04] But would that work with old clients?
[2014-01-03 02:04] it should
[2014-01-03 02:05] to them query string is just noise
[2014-01-03 02:07] okay that sounds reasonable enough for me
[2014-01-03 02:07] What about you Pretorian?
[2014-01-03 02:10] Send the same additional string to all hubs? I’m OK with that.
[2014-01-03 02:10] not some additional string the QS for that hub
[2014-01-03 02:10] (query string)
[2014-01-03 02:10] because hub has no access to it presently