Case of study I: Testing issue in RCS clients (part I)

The reasons why a user isn’t definitively exploding a service can be very diverse. Several features like visual design, accessible interface, simplicity or additional value offered, would help a user decide in favor of RCS. However we believe the main points why RCS is not triumphing among the users are the unacceptable glitches often faced.

Therefore in our blog we will pay special attention to explore the malfunctions of RCS. The aim will be the improvement of the performance of the service, to help refining an enriched messaging service.

As a starting point we will have a look at one important testing issue noticed while executing our GCF RCS 5.1 (Blackbird) test scope. The Global Certification Forum is an independent certification scheme for mobile phones and wireless devices that are based on 3GPP standards. With the definition of an independent certification program, it helps to ensure global interoperability between mobile devices and networks. The GCF certification process is based on technical requirements as specified within dedicated test specifications provided by the 3GPP, 3GPP2, the GSM Association and others.

In the case that concerns us, GCF RCS certification scheme is based on GSMA test descriptions, as the GSMA has officially been the project ‘home’ of RCS defining a whole bunch of test cases in order to improve the quality of testing, increase transparency, drive scale, minimize complexity and accelerate time-to-market (TTM) of RCS services.

So let’s point directly to the issue under study. It can be figured out when performing the GCF-CC F-5.2 (version 3.61.0) test case:

bC2S-4.2.1           1-to-1 Chat – Session Interrupted (Reference Temporarily loses RCS service).

That matches with the TS11 (version 14.1) test description 58.6.1.9:
http://www.gsma.com/newsroom/wp-content/uploads//TS_11_v14_1.zip

58.6.1.9           1-to-1 Chat – Session Interrupted (Reference Temporarily loses RCS service)

Description
Testing the behaviour when an established chat session is closed by a sudden interruption

Related core specifications
GSMA RCS 5.1- Advanced Communications: Services and Client Specification v4.0 – 3.3.4.1.3
Joyn Blackbird Product Definition Document v4.0 – 5.1.1

Reason for test
DUT indication when abnormal interruption, depending on MNO (IM WARN SF)

Initial configuration
DUT and Reference 1 are registered and able to access IMS/RCS core network and relevant servers
Supported by network: 2G, 3G, HSPA, Wi-Fi
Chat session established between DUT and Reference 1 and several messages have been exchanged

Test Procedure

  1. Reference 1 closes the chat session by any abnormal interruption such as loss of network coverage, client error, IP reconfiguration due to a new data bearer, etc.
  2. Check Reference 1 status on DUT while waiting for Server inactivity timer to timeout so chat session is terminated.
  3. DUT sends a message to Reference 1

Expected behaviour

  1. Inactivity timer in the IM Server is trigged and detects that Reference 1 is no longer available and the chat session is terminated.
  2. If Reference 1 is available or capabilities have changed and File Transfer is not available, the share file button should become greyed out
  3. DUT should not get Delivery/Displayed notifications. As Store and Forward is in place, DUT should receive a message to indicate that further messages will be deferred

To demonstrate the bad network behaviour for this case, we have used the following environment:

  • Network operator A.
    IM store and forward = YES
    <parm name=“imCapAlwaysON“ value=“0″ />
    <parm name=“imWarnSF“ value=“0″ />
  • Phone 1 with RCS Client 1 (Own Manufacturer client).
  • Phone 2 with RCS Client 2 (Operator client from Network A).
  • Phone 3 with RCS Client 3 (Own Manufacturer 2 client).

The test results obtained are shown in the table below:Table_RCS_client_Issue_1

As it can be clearly seen, from the result obtained in the steps 2 and 3 above highlighted in red, three different RCS clients operating in the same RCS network could have completely different behaviours for the end users. Both RCS clients from the phone manufacturers can operate for store & forward, however the Manufacturer’s client 2 is not updating the current active screen when the distant party goes offline and the timer expired, while the Manufacturer’s 1 client does.

Also both Manufacturers’ clients, need user interaction to change to send chats (in case of Manufacturer’s 1 client, some client functionality seems to be disturbing the user’s choice, hence when the user doesn’t notice the client itself goes to SMS mode, it could entail SMS pricing charges). In the other hand the RCS client from the operator is not even able to perform store & forward after chat inactivity timer is expired, therefore this operator client is limiting its own network supported features.

From testing perspective according GSMA TS11, test 1 is not fulfilling step 3 expected behaviour. Moreover there is an annoying client automatic motion from chat mode to SMS mode, clearly disturbing the user. Although this last behaviour is not enough criteria to FAIL the test case, as is not contemplated in the specifications, the missing notification indicating further messages will be deferred makes definitively setting the test result to FAIL.

In the case of test 2, step 3 is not possible to be executed because of client limitation, therefore the test result would be N/A (not applicable).

In the case of test attempt 3, the test could be considered PASS with some user/tester interaction in step 2, as this additional step performed by the tester going out and in of the chat screen is not considered in the test expected behaviour. Testing strictly and noticing that the other two clients are able to change to SMS mode in the active chat screen, we could consider also a FAIL result and we would have given a correct test verdict as well.

In summary, we have three test results, which combined, are showing that the proper functioning could be achieved, nevertheless single results show the potential improvement of the clients’ performance.

But why at this late stage of RCS development are we having so different results between three different RCS clients and none of them are really working correctly?

If you can shed light into this darkness, please leave us a comment.

Nie wieder einen Blogartikel verpassen >>> Zum Blog-Newsletter anmelden

Schreibe einen Kommentar

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert.

neunzehn − achtzehn =